U 9%e@sdZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#d d l$m%Z%e"&e'Z(dZ)dZ*d ddgZ+dZ,dZ-dgZ.Gddde j/Z0Gddde j/Z1Gddde j/Z2Gddde j/Z3Gddde j/Z4Gddde j/Z5Gd d!d!e j/Z6Gd"d#d#e j/Z7Gd$d%d%e j/Z8Gd&d'd'eZ9d(Z:d)Z;e d*e:Gd+d,d,e9Ze d2e:Gd3d4d4e9Z?dS)5z PyTorch ViT model.N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedImageModelingOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViTConfigrz!google/vit-base-patch16-224-in21kizgoogle/vit-base-patch16-224z Egyptian catcsdeZdZdZd eeddfdd Zeje e ejddd Z deje ej eejd d d Z ZS) ViTEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. FN)configuse_mask_tokenreturncstttdd|j|_|rrOr% isinstance collectionsabcIterabler*rConv2d projection)r/rr[r>rOr%r*r0r2r3r"s  zViTPatchEmbeddings.__init__F)rJrIr c Cs|j\}}}}||jkr0td|jd|d|s~||jdksP||jdkr~td|d|d|jdd|jdd ||d dd }|S) NzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rrzInput image size (*z) doesn't match model (z).r8)r=rO ValueErrorr[raflatten transpose)r/rJrIrNrOr5r6r4r2r2r3rQs (zViTPatchEmbeddings.forward)F) rRrSrTrUr"r$rWrVrQrYr2r2r0r3r(s r(csleZdZeddfdd ZejejdddZd eeje e e ejejfe ejfd d d Z Z S) ViTSelfAttentionNrr cst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rb)bias)r!r"r%num_attention_headshasattrrdr@attention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer,Zattention_probs_dropout_probr.r/rr0r2r3r"s zViTSelfAttention.__init__)xr cCs6|dd|j|jf}||}|ddddS)Nr7rr8rr )sizerjrlrFrC)r/rsZ new_x_shaper2r2r3transpose_for_scoress z%ViTSelfAttention.transpose_for_scoresF) head_maskoutput_attentionsr c Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |dk r| |} t| |} | dddd} | dd|jf} | | } |r| | fn| f} | S)Nr7r:r;rr8rr )rorurprqr$matmulrfrArBrlrrDZsoftmaxr.rC contiguousrtrmrF) r/ hidden_statesrvrwZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr2r2r3rQs      zViTSelfAttention.forward)NF)rRrSrTrr"r$rWrurrVrrrQrYr2r2r0r3rgsrgcs@eZdZdZeddfdd ZejejejdddZZ S) ViTSelfOutputz The residual connection is defined in ViTLayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nrhcs.tt|j|j|_t|j|_dSN) r!r"rrnr%denser,r-r.rrr0r2r3r"s zViTSelfOutput.__init__rz input_tensorr cCs||}||}|Sr}r~r.r/rzrr2r2r3rQs  zViTSelfOutput.forward) rRrSrTrUrr"r$rWrQrYr2r2r0r3r|sr|cspeZdZeddfdd ZeeddddZd ej e ej e e e ej ej fe ej fd d d ZZS) ViTAttentionNrhcs*tt||_t||_t|_dSr})r!r"rg attentionr|outputset pruned_headsrrr0r2r3r"s   zViTAttention.__init__)headsr cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr;)lenrrrjrlrrrorprqrr~rmunion)r/rindexr2r2r3 prune_headss zViTAttention.prune_headsFrzrvrwr cCs4||||}||d|}|f|dd}|S)Nrr)rr)r/rzrvrwZ self_outputsattention_outputr{r2r2r3rQ szViTAttention.forward)NF)rRrSrTrr"rr@rr$rWrrVrrrQrYr2r2r0r3rsrcs8eZdZeddfdd ZejejdddZZS)ViTIntermediateNrhcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr}) r!r"rrnr%intermediate_sizer~r\Z hidden_actstrr intermediate_act_fnrrr0r2r3r"/s   zViTIntermediate.__init__)rzr cCs||}||}|Sr})r~r)r/rzr2r2r3rQ7s  zViTIntermediate.forward rRrSrTrr"r$rWrQrYr2r2r0r3r.srcs<eZdZeddfdd ZejejejdddZZS) ViTOutputNrhcs.tt|j|j|_t|j|_ dSr}) r!r"rrnrr%r~r,r-r.rrr0r2r3r"?s zViTOutput.__init__rcCs ||}||}||}|Sr}rrr2r2r3rQDs  zViTOutput.forwardrr2r2r0r3r>srcs`eZdZdZeddfdd Zd ejeeje e e ejejfe ejfddd Z Z S) ViTLayerz?This corresponds to the Block class in the timm implementation.Nrhcsbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)NrZeps)r!r"Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrr LayerNormr%layer_norm_epslayernorm_beforelayernorm_afterrrr0r2r3r"Ps    zViTLayer.__init__FrcCs`|j||||d}|d}|dd}||}||}||}|||}|f|}|S)Nrwrr)rrrrr)r/rzrvrwZself_attention_outputsrr{Z layer_outputr2r2r3rQZs     zViTLayer.forward)NF)rRrSrTrUrr"r$rWrrVrrrQrYr2r2r0r3rMs rc sNeZdZeddfdd Zd ejeejeeee e e fddd Z Z S) ViTEncoderNrhcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr2)r).0_rr2r3 {sz'ViTEncoder.__init__..F) r!r"rrZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrrr0rr3r"xs  zViTEncoder.__init__FT)rzrvrwoutput_hidden_states return_dictr c s|rdnd}rdnd}t|jD]\}} |r8||f}|dk rH||nd} |jr||jr|fdd} tjj| | || } n | || } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr2csfdd}|S)Ncs|fSr}r2)inputs)modulerwr2r3custom_forwardszIViTEncoder.forward..create_custom_forward..custom_forwardr2)rrr)rr3create_custom_forwardsz1ViTEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSr}r2)rvr2r2r3 sz%ViTEncoder.forward..)last_hidden_staterz attentions) enumeraterrZtrainingr$utils checkpointtupler) r/rzrvrwrrZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr2rr3rQ~s4       zViTEncoder.forward)NFFT)rRrSrTrr"r$rWrrVrrrrQrYr2r2r0r3rws  rc@s\eZdZdZeZdZdZdZddgZ e e j e j e jfddd d Zdeedd d dZdS)ViTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. vitrJTrrN)rr cCst|tjtjfrZtjj|jjt j d|j j d|jj |j_|jdk r|jjnt|tjr|jj|jjdnjt|trtjj|jjt j d|j j d|jj |j_tjj|jjt j d|j j d|jj |j_dS)zInitialize the weightsg)ZmeanZstdNrL)r\rrnr`initZ trunc_normal_weightdatator$Zfloat32rZinitializer_rangedtyperiZzero_rZfill_rr+r&)r/rr2r2r3 _init_weightss8    z ViTPreTrainedModel._init_weightsF)rrqr cCst|tr||_dSr})r\rr)r/rrqr2r2r3_set_gradient_checkpointings z.ViTPreTrainedModel._set_gradient_checkpointing)F)rRrSrTrUr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingZ_no_split_modulesrrrnr`rrrrVrr2r2r2r3rs raE This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViTImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. interpolate_pos_encoding (`bool`, *optional*): Whether to interpolate the pre-trained position encodings. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z]The bare ViT Model transformer outputting raw hidden-states without any specific head on top.c seZdZdeeedfdd ZedddZee e e fd d d d Z e e eeeed eddeejeejeejeeeeeeeeeeefdddZZS)ViTModelTF)radd_pooling_layerrcs\t|||_t||d|_t||_tj|j |j d|_ |rJt |nd|_ |dS)N)rr)r!r"rrr4rencoderrrr%r layernorm ViTPoolerpooler post_init)r/rrrr0r2r3r"s  zViTModel.__init__)r cCs|jjSr})r4r))r/r2r2r3get_input_embeddingsszViTModel.get_input_embeddingsN)heads_to_pruner cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r/rrrr2r2r3 _prune_headsszViTModel._prune_headsZvision)r output_typerZmodalityexpected_outputrJrKrvrwrrIrr cCs |dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|||jj}|jjj j j }|j |kr~| |}|j|||d} |j | ||||d} | d} || } |jdk r|| nd} |s| dk r| | fn| f} | | ddSt| | | j| jdS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rKrI)rvrwrrrr)rZ pooler_outputrzr)rrwruse_return_dictrdZ get_head_maskrr4r)rarrrrrrrrzr)r/rJrKrvrwrrIrZexpected_dtypeZembedding_outputZencoder_outputssequence_output pooled_outputZ head_outputsr2r2r3rQsD   zViTModel.forward)TF)NNNNNNN)rRrSrTrrVr"r(rrr@rrrVIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr$rWrXrrrQrYr2r2r0r3rs8   rcs*eZdZedfdd ZddZZS)rrcs*tt|j|j|_t|_dSr})r!r"rrnr%r~ZTanh activationrrr0r2r3r"cs zViTPooler.__init__cCs(|dddf}||}||}|S)Nr)r~r)r/rzZfirst_token_tensorrr2r2r3rQhs  zViTPooler.forward)rRrSrTrr"rQrYr2r2r0r3rbsraVViT Model with a decoder on top for masked image modeling, as proposed in [SimMIM](https://arxiv.org/abs/2111.09886). Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). c seZdZeddfdd Zeeeee dd e e j e e j e e j e ee ee ee eeeefdddZZS) ViTForMaskedImageModelingNrhcsXt|t|ddd|_ttj|j|jd|j ddt |j|_ | dS)NFT)rrr8r)Z in_channelsZ out_channelsrZ) r!r"rrrZ Sequentialr`r%encoder_striderOZ PixelShuffledecoderrrrr0r2r3r"~s   z"ViTForMaskedImageModeling.__init__)rrrc Cs|dk r |n|jj}|dk rJ|jj|jjkrJtd|jjd|jjd|j|||||||d}|d} | ddddf} | j\} } } t| d} }| dd d | | | |} | | }d}|dk rD|jj |jj}| d ||}| |jjd |jjd d}tjj||d d }|||d |jj}|st|f|dd}|dk rp|f|S|St|||j|jdS)a= bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, ViTForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k") >>> model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction >>> list(reconstructed_pixel_values.shape) [1, 3, 224, 224] ```NzWhen `bool_masked_pos` is provided, `patch_size` must be equal to `encoder_stride` to ensure that the reconstructed image has the same dimensions as the input.Got `patch_size` = z and `encoder_stride` = rb)rKrvrwrrIrrrg?r8r7none)Z reductiongh㈵>)lossZreconstructionrzr)rrr>rrdrr=rAfloorrCr?rr[Zrepeat_interleaverHryrrDZl1_losssumrOrrzr)r/rJrKrvrwrrIrr{rrNZsequence_lengthrOr5r6Zreconstructed_pixel_valuesZmasked_im_lossrtrPZreconstruction_lossrr2r2r3rQsT)     z!ViTForMaskedImageModeling.forward)NNNNNNN)rRrSrTrr"rrrrrrr$rWrXrVrrrQrYr2r2r0r3rqs(   ra ViT Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. c seZdZeddfdd Zeeeee e e dd e e je e je e je ee ee ee eeee fdddZZS) ViTForImageClassificationNrhcsRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). N)rvrwrrIrrrZ regressionZsingle_label_classificationZmulti_label_classificationr7)rlogitsrzr)rrrrrZdeviceZ problem_typerrr$longr@r Zsqueezer rFr rrzr)r/rJrvrrwrrIrr{rrrZloss_fctrr2r2r3rQ sP      "     z!ViTForImageClassification.forward)NNNNNNN)rRrSrTrr"rrr_IMAGE_CLASS_CHECKPOINTrr_IMAGE_CLASS_EXPECTED_OUTPUTrr$rWrVrrrQrYr2r2r0r3rs2  r)@rUcollections.abcr]rAtypingrrrrrrr$Ztorch.utils.checkpointrZtorch.nnr r r Z activationsr Zmodeling_outputsrrrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_vitrZ get_loggerrRloggerrrrrrZ!VIT_PRETRAINED_MODEL_ARCHIVE_LISTModulerr(rgr|rrrrrrZVIT_START_DOCSTRINGrrrrrr2r2r2r3sb       P'<'*9+ _  r