U ,-eʍ@sNdZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!m"Z"m#Z#dd l$m%Z%d dl&m'Z'e"(e)Z*dZ+dZ,d ddgZ-dZ.dgZ/Gddde j0Z1Gddde j0Z2Gddde j0Z3Gddde j0Z4Gddde j0Z5Gddde j0Z6dGd,d-d-e j0Z?Gd.d/d/eZ@d0ZAd1ZBd2ZCe d3eAGd4d5d5e@ZDe d6eAGd7d8d8e@ZEe d9eAGd:d;d;e@e%ZFdS)=z PyTorch DINOv2 model.N)DictListOptionalSetTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BackboneOutputBaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BackboneMixin) Dinov2Configrzfacebook/dinov2-baseiics^eZdZdZeddfdd ZejeeejdddZ d eje ejejd d d Z Z S) Dinov2EmbeddingszM Construct the CLS token, mask token, position and patch embeddings. Nconfigreturncs~tttdd|j|_ttd|j|_ t ||_ |j j }ttd|d|j|_ t|j|_||_dS)Nr)super__init__r ParametertorchZrandn hidden_size cls_tokenZzeros mask_tokenDinov2PatchEmbeddingspatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutr)selfrr* __class__k/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/dinov2/modeling_dinov2.pyr"Gs  zDinov2Embeddings.__init__) embeddingsheightwidthr c CsR|jdd}|jjdd}||kr4||kr4|jS|jdddf}|jddddf}|jd}||jj}||jj}|d|d}}|dtt|tt||}|dddd}t j j ||t||t|fdd d }t||jd kst||jdkr t d |dddd dd|}tj|d|fdd S)a# This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. Source: https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174 rNrg?r ZbicubicF)Z scale_factormodeZ align_cornerszHWidth or height does not match with the interpolated position embeddingsdim)shaper+r patch_sizereshapeintmathsqrtpermuter functionalZ interpolate ValueErrorviewr$cat unsqueeze) r/r4r5r6r*Z num_positionsZclass_pos_embedZpatch_pos_embedr<r2r2r3interpolate_pos_encodingRs,    $(z)Dinov2Embeddings.interpolate_pos_encoding) pixel_valuesbool_masked_posr c Cs|j\}}}}||}|dk rDt|d|j|jd|}|j |dd}tj ||fdd}|| |||}| |}|S)Nr7rrr;) r=r)r$whererHr'todtyper&expandrGrIr.) r/rJrK batch_size_r5r6r4Z cls_tokensr2r2r3forwardts  zDinov2Embeddings.forward)N) __name__ __module__ __qualname____doc__rr"r$Tensorr@rIrrR __classcell__r2r2r0r3rBs "rcs4eZdZdZfddZejejdddZZS)r(z This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr)Z kernel_sizeZstride)r!r" image_sizer> num_channelsr% isinstance collectionsabcIterabler*rConv2d projection)r/rrYr>rZr%r*r0r2r3r"s  zDinov2PatchEmbeddings.__init__)rJr cCsH|jd}||jkr,td|jd|d||ddd}|S)NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .r8)r=rZrEr`flatten transpose)r/rJrZr4r2r2r3rRs  zDinov2PatchEmbeddings.forward) rSrTrUrVr"r$rWrRrXr2r2r0r3r(s r(csleZdZeddfdd ZejejdddZd eeje e e ejejfe ejfd d d Z Z S) Dinov2SelfAttentionNrcst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads rabias)r!r"r%num_attention_headshasattrrEr@attention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer,Zattention_probs_dropout_probr.r/rr0r2r3r"s zDinov2SelfAttention.__init__)xr cCs6|dd|j|jf}||}|ddddS)Nr7rr8rr )sizergrirFrC)r/rpZ new_x_shaper2r2r3transpose_for_scoress z(Dinov2SelfAttention.transpose_for_scoresF) head_maskoutput_attentionsr c Cs||}|||}|||}||}t||dd}|t|j }t j j |dd} | | } |dk r| |} t| |} | dddd} | dd|jf} | | } |r| | fn| f} | S)Nr7r:r;rr8rr )rlrrrmrnr$matmulrcrArBrirrDZsoftmaxr.rC contiguousrqrjrF) r/ hidden_statesrsrtZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr2r2r3rRs      zDinov2SelfAttention.forward)NF)rSrTrUrr"r$rWrrrboolrrrRrXr2r2r0r3rdsrdcs@eZdZdZeddfdd ZejejejdddZZ S) Dinov2SelfOutputz The residual connection is defined in Dinov2Layer instead of here (as is the case with other models), due to the layernorm applied before each block. Nrcs.tt|j|j|_t|j|_dSN) r!r"rrkr%denser,r-r.ror0r2r3r"s zDinov2SelfOutput.__init__)rw input_tensorr cCs||}||}|Sr{)r|r.)r/rwr}r2r2r3rRs  zDinov2SelfOutput.forward) rSrTrUrVrr"r$rWrRrXr2r2r0r3rzsrzcspeZdZeddfdd ZeeddddZd ej e ej e e e ej ej fe ej fd d d ZZS) Dinov2AttentionNrcs*tt||_t||_t|_dSr{)r!r"rd attentionrzoutputset pruned_headsror0r2r3r"s   zDinov2Attention.__init__)headsr cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr;)lenrrrgrirrrlrmrnrr|rjunion)r/rindexr2r2r3 prune_headss zDinov2Attention.prune_headsFrwrsrtr cCs4||||}||d|}|f|dd}|S)Nrr)rr)r/rwrsrtZ self_outputsattention_outputrxr2r2r3rRszDinov2Attention.forward)NF)rSrTrUrr"rr@rr$rWrryrrrRrXr2r2r0r3r~sr~cs6eZdZddfdd ZejejdddZZS)Dinov2LayerScaleNr cs(tt|jt|j|_dSr{) r!r"rr#Zlayerscale_valuer$Zonesr%lambda1ror0r2r3r"#s zDinov2LayerScale.__init__ hidden_stater cCs ||jSr{)rr/rr2r2r3rR'szDinov2LayerScale.forwardrSrTrUr"r$rWrRrXr2r2r0r3r"srF)input drop_probtrainingr cCsd|dks |s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)rNdevice)r=ndimr$ZrandrNrZfloor_div)rrrZ keep_probr=Z random_tensorrr2r2r3 drop_path,s rcsPeZdZdZd eeddfdd ZejejdddZ e d d d Z Z S) Dinov2DropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).N)rr cst||_dSr{)r!r"r)r/rr0r2r3r"Ds zDinov2DropPath.__init__)rwr cCst||j|jSr{)rrr)r/rwr2r2r3rRHszDinov2DropPath.forwardrcCs d|jS)Nzp={})formatrr/r2r2r3 extra_reprKszDinov2DropPath.extra_repr)N) rSrTrUrVrfloatr"r$rWrRstrrrXr2r2r0r3rAsrcs6eZdZddfdd ZejejdddZZS) Dinov2MLPNrcsnt|j}}t|j|j}tj||dd|_t|j t rPt |j |_ n|j |_ tj||dd|_ dS)NTre)r!r"r%r@ mlp_ratiorrkfc1r[Z hidden_actrr activationfc2r/rZ in_features out_featuresZhidden_featuresr0r2r3r"Ps   zDinov2MLP.__init__rcCs"||}||}||}|Sr{)rrrrr2r2r3rR[s   zDinov2MLP.forwardrr2r2r0r3rOs rcs6eZdZddfdd ZejejdddZZS)Dinov2SwiGLUFFNNrcslt|j}}t|j|j}t|ddddd}tj|d|dd|_tj||dd|_dS)Nr8r Tre) r!r"r%r@rrrk weights_in weights_outrr0r2r3r"cs   zDinov2SwiGLUFFN.__init__rcCs6||}|jddd\}}tj||}||S)Nr8r7r;)rchunkrrDZsilur)r/rx1Zx2Zhiddenr2r2r3rRls zDinov2SwiGLUFFN.forwardrr2r2r0r3rbs rcs`eZdZdZeddfdd Zd ejeeje e e ejejfe ejfddd Z Z S) Dinov2LayerzCThis corresponds to the Block class in the original implementation.Nrcsttj|j|jd|_t||_t ||_ |j dkrFt |j nt |_tj|j|jd|_|jrvt||_n t||_t ||_|j dkrt |j nt |_dS)NZepsr)r!r"r LayerNormr%layer_norm_epsnorm1r~rr layer_scale1Zdrop_path_raterIdentityZ drop_path1norm2Zuse_swiglu_ffnrmlpr layer_scale2Z drop_path2ror0r2r3r"vs      zDinov2Layer.__init__FrcCsp|j||||d}|d}||}|dd}||}||}||}||}||}|f|}|S)Nrtrr)rrrrrr)r/rwrsrtZself_attention_outputsrrxZ layer_outputr2r2r3rRs      zDinov2Layer.forward)NF)rSrTrUrVrr"r$rWrryrrrRrXr2r2r0r3rssrc sNeZdZeddfdd Zd ejeejeeee e e fddd Z Z S) Dinov2EncoderNrcs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr2)r.0rQrr2r3 sz*Dinov2Encoder.__init__..F) r!r"rrZ ModuleListrangenum_hidden_layerslayergradient_checkpointingror0rr3r"s  zDinov2Encoder.__init__FT)rwrsrtoutput_hidden_states return_dictr c s|rdnd}rdnd}t|jD]\}} |r8||f}|dk rH||nd} |jr||jr|fdd} tjj| | || } n | || } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr2csfdd}|S)Ncs|fSr{r2)inputs)modulertr2r3custom_forwardszLDinov2Encoder.forward..create_custom_forward..custom_forwardr2)rrr)rr3create_custom_forwardsz4Dinov2Encoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSr{r2)rvr2r2r3 sz(Dinov2Encoder.forward..)last_hidden_staterw attentions) enumeraterrrr$utils checkpointtupler) r/rwrsrtrrZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr2rr3rRs4       zDinov2Encoder.forward)NFFT)rSrTrUrr"r$rWrryrrrrRrXr2r2r0r3rs  rc@sTeZdZdZeZdZdZdZe e j e j e j fddddZd eedd d d ZdS)Dinov2PreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. dinov2rJTN)rr cCst|tjtjfrZtjj|jjt j d|j j d|jj |j_|jdk r|jjnt|tjr|jj|jjdnjt|trtjj|jjt j d|j j d|jj |j_tjj|jjt j d|j j d|jj |j_dS)zInitialize the weightsr)meanZstdNg?)r[rrkr_initZ trunc_normal_weightdatarMr$Zfloat32rZinitializer_rangerNrfZzero_rZfill_rr+r&)r/rr2r2r3 _init_weightss8    z#Dinov2PreTrainedModel._init_weightsF)rrnr cCst|tr||_dSr{)r[rr)r/rrnr2r2r3_set_gradient_checkpointings z1Dinov2PreTrainedModel._set_gradient_checkpointing)F)rSrTrUrVr config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrrkr_rrrryrr2r2r2r3rs raH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Dinov2Config`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a4 Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BitImageProcessor.preprocess`] for details. bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Only relevant for pre-training. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aM Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BitImageProcessor.preprocess`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z`The bare DINOv2 Model transformer outputting raw hidden-states without any specific head on top.c seZdZedfdd ZedddZeee efddd d Z e e e eeed ed deejeejeejeeeeeeeeefd ddZZS) Dinov2ModelrcsFt|||_t||_t||_tj|j |j d|_ | dS)Nr) r!r"rrr4rencoderrrr%r layernorm post_initror0r2r3r"Ls    zDinov2Model.__init__rcCs|jjSr{r4r)rr2r2r3get_input_embeddingsXsz Dinov2Model.get_input_embeddingsN)heads_to_pruner cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r/rrrr2r2r3 _prune_heads[szDinov2Model._prune_headsZvision)r output_typerZmodalityZexpected_output)rJrKrsrtrrr c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|||jj}|j||d}|j|||||d}|d} | | } | dddddf} |s| | f} | |ddSt | | |j |j dS)Nz You have to specify pixel_values)rKrsrtrrrr)rZ pooler_outputrwr) rrtruse_return_dictrEZ get_head_maskrr4rrrrwr) r/rJrKrsrtrrembedding_outputZencoder_outputssequence_outputZ pooled_outputZ head_outputsr2r2r3rRcs6 zDinov2Model.forward)NNNNNN)rSrTrUrr"r(rrr@rrrDINOV2_BASE_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr$rWryrrrRrXr2r2r0r3rGs4   rz Dinov2 Model transformer with an image classification head on top (a linear layer on top of the final hidden state of the [CLS] token) e.g. for ImageNet. c s|eZdZeddfdd Zeeeee e dd e e j e e j e e j e ee ee eeee fdddZZS) Dinov2ForImageClassificationNrcsRt||j|_t||_|jdkr 1` a classification loss is computed (Cross-Entropy). Nrrrr;Z regressionZsingle_label_classificationZmulti_label_classificationr7r8)losslogitsrwr)rrrr$rGrrrMrZ problem_typerrNlongr@r Zsqueezer rFr rrwr)r/rJrsrrtrrrxrr&Z patch_tokensZ linear_inputrrZloss_fctrr2r2r3rRsT      "    z$Dinov2ForImageClassification.forward)NNNNNN)rSrTrUrr"rDINOV2_INPUTS_DOCSTRINGr_IMAGE_CLASS_CHECKPOINTrrrr$rWryrrrRrXr2r2r0r3rs, rzO Dinov2 backbone, to be used with frameworks like DETR and MaskFormer. c sdeZdZfddZedddZeeee e dd e j e ee ee ee dd d ZZS) Dinov2BackbonecsjttfddtjdD|_t|_t|_ t j j j d|_|dS)Ncsg|] }jqSr2)r%rrr2r3r sz+Dinov2Backbone.__init__..rr)r!r"Z_init_backbonerrZ num_featuresrr4rrrrr%rrrror0rr3r"s    zDinov2Backbone.__init__rcCs|jjSr{rrr2r2r3rsz#Dinov2Backbone.get_input_embeddings)rrN)rJrrtrr cCsh|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}||}|j|d||d}|rb|jn|d}d}t|j|D]\} } | |j krz|jj r| | } |jj r|j \} } } }|jj}| ddddddf| ||| |d} | dddd } || f7}qz|sB|r,|f|dd}n|f|d d}|St||rR|jnd|r`|jndd S) a7 Returns: Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base") >>> model = AutoBackbone.from_pretrained( ... "facebook/dinov2-base", out_features=["stage2", "stage5", "stage8", "stage11"] ... ) >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 16, 16] ```NT)rrtrrr2r7rr r8) feature_mapsrwr)rrrrtr4rrwzipZ stage_namesrZapply_layernormrZreshape_hidden_statesr=r>r?rCrvrr)r/rJrrtrrrxrwrZstagerrPrQr5r6r>rr2r2r3rRsL#     zDinov2Backbone.forward)NNN)rSrTrUr"r(rrrrrrr$rWrryrRrXr2r2r0r3rs  r)rF)GrVcollections.abcr\rAtypingrrrrrrr$Ztorch.utils.checkpointrZtorch.nnr r r Z activationsr Zmodeling_outputsrrrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZutils.backbone_utilsrZconfiguration_dinov2rZ get_loggerrSloggerrrrrZ$DINOV2_PRETRAINED_MODEL_ARCHIVE_LISTModulerr(rdrzr~rrWrryrrrrrrrZDINOV2_START_DOCSTRINGrrrrrr2r2r2r3sh        G"=' 59* R\