U 9%e@sJdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&e#'e(Z)dZ*dZ+d ddgZ,dZ-dZ.dgZ/Gddde j0Z1Gddde j0Z2Gddde j0Z3Gddde j0Z4Gddde j0Z5Gddde j0Z6Gd d!d!e j0Z7Gd"d#d#e j0Z8Gd$d%d%e j0Z9Gd&d'd'eZ:d(Z;d)Ze!d/e;Gd0d1d1e:Z?e!d2e;Gd3d4d4e:Z@eGd5d6d6eZAe!d7e;Gd8d9d9e:ZBdS):z PyTorch DeiT model.N) dataclass)OptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) DeiTConfigrz(facebook/deit-base-distilled-patch16-224iztabby, tabby catcsJeZdZdZd eeddfdd Zd eje ej ejddd Z Z S) DeiTEmbeddingszv Construct the CLS token, distillation token, position and patch embeddings. Optionally, also the mask token. FN)configuse_mask_tokenreturncstttdd|j|_ttdd|j|_|rTttdd|jnd|_ t ||_ |j j }ttd|d|j|_ t|j|_dS)Nr)super__init__r ParametertorchZzeros hidden_size cls_tokendistillation_token mask_tokenDeiTPatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropout)selfrrr, __class__e/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/deit/modeling_deit.pyr#Is   zDeiTEmbeddings.__init__) pixel_valuesbool_masked_posr c Cs||}|\}}}|dk rT|j||d}|d|}|d|||}|j|dd} |j|dd} tj | | |fdd}||j }| |}|S)N?rdim) r+sizer)expand unsqueezeZtype_asr'r(r%catr-r0) r1r6r7 embeddings batch_sizeZ seq_length_Z mask_tokensmaskZ cls_tokensZdistillation_tokensr4r4r5forwardTs   zDeiTEmbeddings.forward)F)N) __name__ __module__ __qualname____doc__rboolr#r%Tensorr BoolTensorrD __classcell__r4r4r2r5rDs rcs4eZdZdZfddZejejdddZZS)r*z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr) kernel_sizeZstride)r"r# image_size patch_size num_channelsr& isinstance collectionsabcIterabler,rConv2d projection)r1rrNrOrPr&r,r2r4r5r#ms  zDeiTPatchEmbeddings.__init__)r6r c Cs|j\}}}}||jkr td||jdks<||jdkrjtd|d|d|jdd|jdd ||ddd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).r!)shaperP ValueErrorrNrVflatten transpose)r1r6rArPheightwidthxr4r4r5rD|s (zDeiTPatchEmbeddings.forward) rErFrGrHr#r%rJrDrLr4r4r2r5r*fs r*csleZdZeddfdd ZejejdddZd eeje e e ejejfe ejfd d d Z Z S) DeiTSelfAttentionNrr cst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .)bias)r"r#r&num_attention_headshasattrrYintattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer.Zattention_probs_dropout_probr0r1rr2r4r5r#s zDeiTSelfAttention.__init__)r^r cCs6|dd|j|jf}||}|ddddS)Nr8rr!rr )r<rcrfviewpermute)r1r^Z new_x_shaper4r4r5transpose_for_scoress z&DeiTSelfAttention.transpose_for_scoresF) head_maskoutput_attentionsr c Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |dk r| |} t| |} | dddd} | dd|jf} | | } |r| | fn| f} | S)Nr8r:rr!rr )rirorjrkr%matmulr[mathsqrtrfr functionalZsoftmaxr0rn contiguousr<rgrm) r1 hidden_statesrprqZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr4r4r5rDs      zDeiTSelfAttention.forward)NF)rErFrGrr#r%rJrorrIrrrDrLr4r4r2r5r_sr_cs@eZdZdZeddfdd ZejejejdddZZ S) DeiTSelfOutputz The residual connection is defined in DeiTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nr`cs.tt|j|j|_t|j|_dSN) r"r#rrhr&denser.r/r0rlr2r4r5r#s zDeiTSelfOutput.__init__rx input_tensorr cCs||}||}|Sr{r|r0r1rxr~r4r4r5rDs  zDeiTSelfOutput.forward) rErFrGrHrr#r%rJrDrLr4r4r2r5rzsrzcspeZdZeddfdd ZeeddddZd ej e ej e e e ej ej fe ej fd d d ZZS) DeiTAttentionNr`cs*tt||_t||_t|_dSr{)r"r#r_ attentionrzoutputset pruned_headsrlr2r4r5r#s   zDeiTAttention.__init__)headsr cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr:)lenrrrcrfrrrirjrkrr|rgunion)r1rindexr4r4r5 prune_headss zDeiTAttention.prune_headsFrxrprqr cCs4||||}||d|}|f|dd}|S)Nrr)rr)r1rxrprqZ self_outputsattention_outputryr4r4r5rDszDeiTAttention.forward)NF)rErFrGrr#rrerr%rJrrIrrrDrLr4r4r2r5rsrcs8eZdZeddfdd ZejejdddZZS)DeiTIntermediateNr`csBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr{) r"r#rrhr&intermediate_sizer|rQZ hidden_actstrr intermediate_act_fnrlr2r4r5r#s   zDeiTIntermediate.__init__)rxr cCs||}||}|Sr{)r|r)r1rxr4r4r5rD s  zDeiTIntermediate.forward rErFrGrr#r%rJrDrLr4r4r2r5rsrcs<eZdZeddfdd ZejejejdddZZS) DeiTOutputNr`cs.tt|j|j|_t|j|_ dSr{) r"r#rrhrr&r|r.r/r0rlr2r4r5r#s zDeiTOutput.__init__r}cCs ||}||}||}|Sr{rrr4r4r5rDs  zDeiTOutput.forwardrr4r4r2r5rsrcs`eZdZdZeddfdd Zd ejeeje e e ejejfe ejfddd Z Z S) DeiTLayerz?This corresponds to the Block class in the timm implementation.Nr`csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)NrZeps)r"r#Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr&layer_norm_epslayernorm_beforelayernorm_afterrlr2r4r5r#'s    zDeiTLayer.__init__FrcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)Nrqrr)rrrrr)r1rxrprqZself_attention_outputsrryZ layer_outputr4r4r5rD1s     zDeiTLayer.forward)NF)rErFrGrHrr#r%rJrrIrrrDrLr4r4r2r5r$s rc sNeZdZeddfdd Zd ejeejeeee e e fddd Z Z S) DeiTEncoderNr`cs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr4)r).0rBrr4r5 Ssz(DeiTEncoder.__init__..F) r"r#rrZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrlr2rr5r#Ps  zDeiTEncoder.__init__FT)rxrprqoutput_hidden_states return_dictr c s|rdnd}rdnd}t|jD]\}} |r8||f}|dk rH||nd} |jr||jr|fdd} tjj| | || } n | || } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr4csfdd}|S)Ncs|fSr{r4)inputs)modulerqr4r5custom_forwardjszJDeiTEncoder.forward..create_custom_forward..custom_forwardr4)rrr)rr5create_custom_forwardisz2DeiTEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSr{r4)rvr4r4r5 sz&DeiTEncoder.forward..)last_hidden_staterx attentions) enumeraterrZtrainingr%utils checkpointtupler ) r1rxrprqrrZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr4rr5rDVs4       zDeiTEncoder.forward)NFFT)rErFrGrr#r%rJrrIrrr rDrLr4r4r2r5rOs  rc@sZeZdZdZeZdZdZdZdgZ e e j e j e jfdddd Zdeedd d d ZdS)DeiTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. deitr6TrN)rr cCst|tjtjfrZtjj|jjt j d|j j d|jj |j_|jdk r|jjn&t|tjr|jj|jjddS)zInitialize the weightsg)ZmeanZstdNr9)rQrrhrUinitZ trunc_normal_weightdatator%Zfloat32rZinitializer_rangedtyperbZzero_rZfill_)r1rr4r4r5 _init_weightss   z!DeiTPreTrainedModel._init_weightsF)rrkr cCst|tr||_dSr{)rQrr)r1rrkr4r4r5_set_gradient_checkpointings z/DeiTPreTrainedModel._set_gradient_checkpointing)F)rErFrGrHr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesrrrhrUrrrrIrr4r4r4r5rs raF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`DeiTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aL Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`DeiTImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare DeiT Model transformer outputting raw hidden-states without any specific head on top.c seZdZdeeeddfdd Zeddd Zd d Ze e e e e ed ed deejeejeejeeeeeeeee fdddZZS) DeiTModelTFN)radd_pooling_layerrr cs\t|||_t||d|_t||_tj|j |j d|_ |rJt |nd|_ |dS)N)rr)r"r#rrr@rencoderrrr&r layernorm DeiTPoolerpooler post_init)r1rrrr2r4r5r#s  zDeiTModel.__init__)r cCs|jjSr{)r@r+)r1r4r4r5get_input_embeddingsszDeiTModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r1Zheads_to_prunerrr4r4r5 _prune_headsszDeiTModel._prune_headsZvision)r output_typerZmodalityexpected_outputr6r7rprqrrr c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|||jj}|jjj j j }|j |kr~| |}|j||d}|j |||||d} | d} || } |jdk r|| nd} |s| dk r| | fn| f} | | ddSt| | | j| jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)r7rprqrrrr)rZ pooler_outputrxr)rrqruse_return_dictrYZ get_head_maskrr@r+rVrrrrrrrrxr) r1r6r7rprqrrZexpected_dtypeZembedding_outputZencoder_outputssequence_output pooled_outputZ head_outputsr4r4r5rDs<   zDeiTModel.forward)TF)NNNNNN)rErFrGrrIr#r*rrrDEIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr%rJrKrrrDrLr4r4r2r5rs4   rcs*eZdZedfdd ZddZZS)rrcs*tt|j|j|_t|_dSr{)r"r#rrhr&r|ZTanh activationrlr2r4r5r#+s zDeiTPooler.__init__cCs(|dddf}||}||}|S)Nr)r|r)r1rxZfirst_token_tensorrr4r4r5rD0s  zDeiTPooler.forward)rErFrGrr#rDrLr4r4r2r5r*sraWDeiT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://arxiv.org/abs/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). c szeZdZeddfdd Zeeeee dd e e j e e j e e j e ee ee eeeefdddZZS) DeiTForMaskedImageModelingNr`csXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr!r)Z in_channelsZ out_channelsrM) r"r#rrrZ SequentialrUr&Zencoder_striderPZ PixelShuffledecoderrrlr2r4r5r#Fs   z#DeiTForMaskedImageModeling.__init__rrrcCsN|dk r |n|jj}|j||||||d}|d}|ddddf}|j\} } } t| d} } |ddd| | | | }||}d}|dk r |jj|jj }|d||}| |jj d |jj d d }t jj||dd }|||d |jj}|s:|f|dd}|dk r6|f|S|St|||j|jd S) aM bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForMaskedImageModeling.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```N)r7rprqrrrrr8g?r!none)Z reductiongh㈵>)lossZreconstructionrxr)rrrrXrernZreshaperrNrOZrepeat_interleaver>rwrrvZl1_losssumrPrrxr)r1r6r7rprqrrryrrAZsequence_lengthrPr\r]Zreconstructed_pixel_valuesZmasked_im_lossr<rCZreconstruction_lossrr4r4r5rDWsJ(     z"DeiTForMaskedImageModeling.forward)NNNNNN)rErFrGrr#rrrrrrr%rJrKrIrrrDrLr4r4r2r5r9s$   rz DeiT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. c szeZdZeddfdd Zeeeee dd e e j e e j e e j e e e e e e eeefdddZZS) DeiTForImageClassificationNr`csRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, DeiTForImageClassification >>> import torch >>> from PIL import Image >>> import requests >>> torch.manual_seed(3) # doctest: +IGNORE_RESULT >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> # note: we are loading a DeiTForImageClassificationWithTeacher from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> image_processor = AutoImageProcessor.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> model = DeiTForImageClassification.from_pretrained("facebook/deit-base-distilled-patch16-224") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> # model predicts one of the 1000 ImageNet classes >>> predicted_class_idx = logits.argmax(-1).item() >>> print("Predicted class:", model.config.id2label[predicted_class_idx]) Predicted class: magpie ```NrrrZ regressionZsingle_label_classificationZmulti_label_classificationr8)rlogitsrxr)rrrrrZdeviceZ problem_typerrr%longrer Zsqueezer rmrrrxr) r1r6rprrqrrryrrrZloss_fctrr4r4r5rDsN,     "     z"DeiTForImageClassification.forward)NNNNNN)rErFrGrr#rrrrrrr%rJrIrrrDrLr4r4r2r5rs$   rc@sheZdZUdZdZejed<dZejed<dZ ejed<dZ e e ejed<dZ e e ejed<dS)+DeiTForImageClassificationWithTeacherOutputa5 Output type of [`DeiTForImageClassificationWithTeacher`]. Args: logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores as the average of the cls_logits and distillation logits. cls_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the classification head (i.e. the linear layer on top of the final hidden state of the class token). distillation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Prediction scores of the distillation head (i.e. the linear layer on top of the final hidden state of the distillation token). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr cls_logitsdistillation_logitsrxr)rErFrGrHrr%Z FloatTensor__annotations__rrrxrrrr4r4r4r5r s ra DeiT Model transformer with image classification heads on top (a linear layer on top of the final hidden state of the [CLS] token and a linear layer on top of the final hidden state of the distillation token) e.g. for ImageNet. .. warning:: This model supports inference-only. Fine-tuning with distillation (i.e. with a teacher) is not yet supported. c sveZdZeddfdd Zeeeee e e dd e e je e je ee ee eeee fdddZZS) %DeiTForImageClassificationWithTeacherNr`csvt||j|_t|dd|_|jdkrsr        "%=(+9 ]  ik