U ,-e@sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%d dl&m'Z'e$(e)Z*dZ+dZ,d ddgZ-dZ.dZ/dgZ0eGdddeZ1dNe j2e3e4e j2dddZ5Gddde j6Z7Gddde j6Z8Gd d!d!e j6Z9Gd"d#d#e j6Z:Gd$d%d%e j6Z;Gd&d'd'e j6ZGd,d-d-e j6Z?Gd.d/d/e j6Z@Gd0d1d1e j6ZAGd2d3d3eZBd4ZCd5ZDe"d6eCGd7d8d8eBZEGd9d:d:e j6ZFe"d;eCGdeCGd?d@d@eBZHGdAdBdBe j6ZIGdCdDdDe j6ZJGdEdFdFe j6ZKGdGdHdHe j6ZLGdIdJdJe j6ZMe"dKeCGdLdMdMeBZNdS)Oz PyTorch BEiT model.N) dataclass)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutputMaskedLMOutputSemanticSegmenterOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) BeitConfigrz%microsoft/beit-base-patch16-224-pt22kizmicrosoft/beit-base-patch16-224ztabby, tabby catc@seZdZdZdS)BeitModelOutputWithPoolinga Class for outputs of [`BeitModel`]. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N)__name__ __module__ __qualname____doc__r#r#g/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/beit/modeling_beit.pyrCsrF)input drop_probtrainingreturncCsd|dks |s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. r%rrr)dtypedevice)shapendimtorchZrandr+r,Zfloor_div)r&r'r(Z keep_probr-Z random_tensoroutputr#r#r$ drop_path]s r2csPeZdZdZd eeddfdd ZejejdddZ e d d d Z Z S) BeitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).N)r'r)cst||_dSN)super__init__r')selfr' __class__r#r$r6ts zBeitDropPath.__init__ hidden_statesr)cCst||j|jSr4)r2r'r(r7r;r#r#r$forwardxszBeitDropPath.forwardr)cCs d|jS)Nzp={})formatr'r7r#r#r$ extra_repr{szBeitDropPath.extra_repr)N) rr r!r"rfloatr6r/Tensorr=strrA __classcell__r#r#r8r$r3qsr3csFeZdZdZeddfdd Zd ejeej ejdddZ Z S) BeitEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. Nconfigr)cstttdd|j|_|jrBttdd|j|_ nd|_ t ||_ |j j }|j r~ttd|d|j|_nd|_t|j|_dS)Nr)r5r6r Parameterr/zeros hidden_size cls_tokenZuse_mask_token mask_tokenBeitPatchEmbeddingspatch_embeddings num_patchesZ use_absolute_position_embeddingsposition_embeddingsDropouthidden_dropout_probdropout)r7rHrPr8r#r$r6s  zBeitEmbeddings.__init__) pixel_valuesbool_masked_posr)c Cs||}|\}}}|j|dd}|dk rd|j||d}|d|} |d| || }tj||fdd}|j dk r||j }| |}|SNrdim) rOsizerLexpandrM unsqueezeZtype_asr/catrQrT) r7rUrV embeddings batch_sizeZseq_len_Z cls_tokensZ mask_tokenswr#r#r$r=s    zBeitEmbeddings.forward)N) rr r!r"rr6r/rCr BoolTensorr=rEr#r#r8r$rFsrFcs4eZdZdZfddZejejdddZZS)rNz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}|d|d|d|df}||_||_||_||_ ||_ t j ||||d|_dS)Nrr kernel_sizeZstride)r5r6 image_size patch_size num_channelsrK isinstance collectionsabcIterablerP patch_shaperConv2d projection)r7rHrfrgrhrKrPrmr8r#r$r6s   zBeitPatchEmbeddings.__init__)rUr)c Cs|j\}}}}||jkr td||jdks<||jdkrjtd|d|d|jdd|jdd ||ddd}|S) NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrzInput image size (*z) doesn't match model (z).)r-rh ValueErrorrfroflatten transpose)r7rUr`rhheightwidthr_r#r#r$r=s (zBeitPatchEmbeddings.forward) rr r!r"r6r/rCr=rEr#r#r8r$rNs rNc sreZdZd eeeddfdd ZddZd ej eej e ede e ej e ej ej ffd d d Z ZS)BeitSelfAttentionNrH window_sizer)cst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t j |j|jdd|_ t |j|j|_ t |j|_|rt||d|_nd|_dS) NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .F)biasry)r5r6rKnum_attention_headshasattrrrintattention_head_size all_head_sizerLinearquerykeyvaluerRZattention_probs_dropout_probrTBeitRelativePositionBiasrelative_position_biasr7rHryr8r#r$r6s zBeitSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrXrrqrr )r[r}rviewpermute)r7xZ new_x_shaper#r#r$transpose_for_scoress z&BeitSelfAttention.transpose_for_scoresFrr; head_maskoutput_attentionsrr)cCs||}|||}|||}||}t||dd} | t|j } |j dk rt| | d} |dk r| |} t j j| dd} || } |dk r| |} t| |} | dddd} | dd|jf} | j| } |r| | fn| f} | S)NrXrrYrqrr )rrrrr/matmulrtmathsqrtrrr]r functionalZsoftmaxrTr contiguousr[rr)r7r;rrrZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr#r#r$r=s(      zBeitSelfAttention.forward)N)NFN)rr r!rrtupler6rr/rCboolrrr=rEr#r#r8r$rwsrwcsBeZdZdZeddfdd Zd ejejejdddZZ S) BeitSelfOutputz The residual connection is defined in BeitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. NrGcs.tt|j|j|_t|j|_dSr4) r5r6rrrKdenserRrSrTr7rHr8r#r$r6$s zBeitSelfOutput.__init__)r; input_tensorr)cCs||}||}|Sr4rrT)r7r;rgammar#r#r$r=)s  zBeitSelfOutput.forward)N) rr r!r"rr6r/rCr=rEr#r#r8r$rsrc sreZdZd eeeddfdd ZddZd ej eej e ede e ej e ej ej ffd d d Z ZS) BeitAttentionNrxcs.tt||d|_t||_t|_dS)Nr|)r5r6rw attentionrr1set pruned_headsrr8r#r$r61s  zBeitAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrY)lenrrr}rrrrrrr1rrunion)r7headsindexr#r#r$ prune_heads7s zBeitAttention.prune_headsFrrcCs6|||||}||d|}|f|dd}|S)Nrr)rr1)r7r;rrrZ self_outputsattention_outputrr#r#r$r=IszBeitAttention.forward)N)NFN)rr r!rrrr6rr/rCrrrr=rEr#r#r8r$r0srcs8eZdZeddfdd ZejejdddZZS)BeitIntermediateNrGcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr4) r5r6rrrKintermediate_sizerriZ hidden_actrDr intermediate_act_fnrr8r#r$r6Ys   zBeitIntermediate.__init__r:cCs||}||}|Sr4)rrr<r#r#r$r=as  zBeitIntermediate.forward rr r!rr6r/rCr=rEr#r#r8r$rXsrcs8eZdZeddfdd ZejejdddZZS) BeitOutputNrGcs.tt|j|j|_t|j|_ dSr4) r5r6rrrrKrrRrSrTrr8r#r$r6is zBeitOutput.__init__r:cCs||}||}|Sr4rr<r#r#r$r=ns  zBeitOutput.forwardrr#r#r8r$rhsrc speZdZdZd eeeeddfdd Zd e j ee j e ede e e j e e j e j ffd d d ZZS) BeitLayerz?This corresponds to the Block class in the timm implementation.Nr%)rHrydrop_path_rater)cst|j|_d|_t||d|_t||_t||_ t j |j |j d|_|dkr^t|nt |_t j |j |j d|_|j}|dkrt j|t|j dd|_t j|t|j dd|_n d\|_|_dS) Nrr|Zepsr%rT)Z requires_grad)NN)r5r6Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterr1r LayerNormrKlayer_norm_epslayernorm_beforer3Identityr2layernorm_afterZlayer_scale_init_valuerIr/Zoneslambda_1lambda_2)r7rHryrZ init_valuesr8r#r$r6xs   zBeitLayer.__init__Frrc Cs|j|||||d}|d}|dd}|jdk r@|j|}|||}||}||}||}|jdk r|j|}|||}|f|}|S)N)rrrr)rrrr2rrr1r) r7r;rrrZself_attention_outputsrrZ layer_outputr#r#r$r=s&         zBeitLayer.forward)Nr%)NFN)rr r!r"rrrrBr6r/rCrrrr=rEr#r#r8r$rusrcs6eZdZeeddfdd ZejdddZZ S)rNrxc st||_d|ddd|ddd|_tt|j|j|_ t |d}t |d}t t ||gdd}t |d}|dddddf|dddddf}|ddd}|dddddf|dd7<|dddddf|dd7<|dddddfd|dd9<tj|d|ddfd|jd}|d|ddddf<|jd|dddf<|jd|dddf<|jd|d <|jd |d d dS) Nrqrrr Zij)Zindexing)r[r+rX)rrrelative_position_indexF) persistent)r5r6ryZnum_relative_distancerrIr/rJr}relative_position_bias_tableZarangestackrrsrrr+sumZregister_buffer) r7rHryZcoords_hZcoords_wZcoordsZcoords_flattenZrelative_coordsrr8r#r$r6s0 & ,&&*z!BeitRelativePositionBias.__init__r>cCsV|j|jd|jd|jdd|jd|jddd}|dddS)NrXrrrq)rrrryrr)r7rr#r#r$r=s z BeitRelativePositionBias.forward) rr r!rrr6r/rCr=rEr#r#r8r$rsrc sVeZdZd eeeddfdd Zd ejeeje e e e ee fddd Z Z S) BeitEncoderNrxcsvt|_jr&td|_nd|_ddtdjj Dt fddt j D|_ d|_dS)Nr|cSsg|] }|qSr#)item.0rr#r#r$ sz(BeitEncoder.__init__..rcs(g|] }tjrnd|dqS)N)ryr)rZuse_relative_position_biasrirHZdprryr#r$rs  F)r5r6rHZ!use_shared_relative_position_biasrrr/Zlinspacernum_hidden_layersr ModuleListrangelayergradient_checkpointingrr8rr$r6s  zBeitEncoder.__init__FT)r;rroutput_hidden_states return_dictr)cs|rdnd}rdnd}t|jD]\}} |r8||f}|dk rH||nd} |jr||jr|fdd} tjj| | || } n$|jdk r|nd} | || | } | d}r"|| df}q"|r||f}|stdd|||fDSt |||dS) Nr#csfdd}|S)Ncs|fSr4r#)inputs)modulerr#r$custom_forwardszJBeitEncoder.forward..create_custom_forward..custom_forwardr#)rrr)rr$create_custom_forwardsz2BeitEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSr4r#)rvr#r#r$ sz&BeitEncoder.forward..)last_hidden_stater; attentions) enumeraterrr(r/utils checkpointrrr )r7r;rrrrZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskrZ layer_outputsrr#rr$r=s8      zBeitEncoder.forward)N)NFFT)rr r!rrrr6r/rCrrr r=rEr#r#r8r$rs rc@s2eZdZdZeZdZdZdZddZ d dd Z d S) BeitPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. beitrUTcCst|tjtjtjfrD|jjjd|jj d|j dk r|j j nft|tj r|jjjd|jj d|j dk r|jj|j  n&t|tjr|j j |jjddS)zInitialize the weightsr%)meanZstdNg?)rirrrnConvTranspose2dweightdataZnormal_rHZinitializer_ranger{Zzero_Z EmbeddingZ padding_idxrZfill_)r7rr#r#r$ _init_weights/s     z!BeitPreTrainedModel._init_weightsFcCst|tr||_dSr4)rirr)r7rrr#r#r$_set_gradient_checkpointing?s z/BeitPreTrainedModel._set_gradient_checkpointingN)F) rr r!r"r config_classZbase_model_prefixZmain_input_nameZsupports_gradient_checkpointingrrr#r#r#r$r$sraF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`BeitConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. aL Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`BeitImageProcessor.__call__`] for details. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare Beit Model transformer outputting raw hidden-states without any specific head on top.c seZdZdeeddfdd ZddZdd Zee e e e e d ed deejeejeejeeeeeeeee fd d dZZS) BeitModelTN)rHadd_pooling_layerr)cspt|||_t||_t||jjjd|_|j r>t nt j |j |jd|_|r^t|nd|_|dS)Nr|r)r5r6rHrFr_rrOrmencoderuse_mean_poolingrrrrKr layernorm BeitPoolerpooler post_init)r7rHrr8r#r$r6ks  zBeitModel.__init__cCs|jjSr4)r_rOr@r#r#r$get_input_embeddingszszBeitModel.get_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r7Zheads_to_prunerrr#r#r$ _prune_heads}szBeitModel._prune_headsZvision)r output_typerZmodalityexpected_output)rUrVrrrrr)c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|||jj}|||}|j|||||d}|d} | | } |j dk r| | nd} |s| dk r| | fn| f} | |ddSt | | |j |j dS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_valuesrrrrrr)r pooler_outputr;r)rHrruse_return_dictrrZ get_head_maskrr_rrrrr;r) r7rUrVrrrrZembedding_outputZencoder_outputssequence_output pooled_outputZ head_outputsr#r#r$r=s6  zBeitModel.forward)T)NNNNNN)rr r!rrr6rrrBEIT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOC_EXPECTED_OUTPUT_SHAPErr/rCrcrrr=rEr#r#r8r$rfs4  rcs8eZdZeddfdd ZejejdddZZS)rNrGcs,t|jr"tj|j|jdnd|_dS)Nr)r5r6rrrrKrrrr8r#r$r6s zBeitPooler.__init__r:cCsJ|jdk r6|ddddddf}||d}n|dddf}|S)Nrr)rr)r7r;Z patch_tokensrr#r#r$r=s  zBeitPooler.forwardrr#r#r8r$rsraBeit Model transformer with a 'language' modeling head on top. BEiT does masked image modeling by predicting visual tokens of a Vector-Quantize Variational Autoencoder (VQ-VAE), whereas other vision models like ViT and DeiT predict RGB pixel values. As a result, this class is incompatible with [`AutoModelForMaskedImageModeling`], so you will need to use [`BeitForMaskedImageModeling`] directly if you wish to do masked image modeling with BEiT.c seZdZeddfdd Zeeeee dd e e j e e j e e j e e j e ee ee eeeefdddZZS) BeitForMaskedImageModelingNrGcsTt||j|_t|dd|_tj|j|jd|_ t |j|j |_ | dS)NFrr)r5r6 num_labelsrrrrrKrrrZ vocab_sizelm_headrrr8r#r$r6s  z#BeitForMaskedImageModeling.__init__rr)rUrVrlabelsrrrr)cCs|dk r |n|jj}|j||||||d}|d} || } || ddddf} d} |dk rvt} | | ||} |s| f|dd} | dk r| f| S| St| | |j|jdS)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, BeitForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> model = BeitForMaskedImageModeling.from_pretrained("microsoft/beit-base-patch16-224-pt22k") >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, logits = outputs.loss, outputs.logits >>> list(logits.shape) [1, 196, 8192] ```N)rVrrrrrrlosslogitsr;r) rHrrrrr rr;r)r7rUrVrrrrrrrZprediction_scoresZmasked_lm_lossloss_fctr1r#r#r$r=s2/  z"BeitForMaskedImageModeling.forward)NNNNNNN)rr r!rr6rrrrrrr/rCrcrrrr=rEr#r#r8r$rs(   rz Beit Model transformer with an image classification head on top (a linear layer on top of the average of the final hidden states of the patch tokens) e.g. for ImageNet. c s~eZdZeddfdd Zeeeee e e dd e e je e je e je ee ee eeee fdddZZS) BeitForImageClassificationNrGcsRt||j|_t|dd|_|jdkr 1` a classification loss is computed (Cross-Entropy). NrrZ regressionZsingle_label_classificationZmulti_label_classificationrXrqr)rHrrrrZ problem_typerr+r/longrr Zsqueezer rrrr;r) r7rUrrrrrrrrrrr1r#r#r$r=NsL     "     z"BeitForImageClassification.forward)NNNNNN)rr r!rr6rrr_IMAGE_CLASS_CHECKPOINTrr_IMAGE_CLASS_EXPECTED_OUTPUTrr/rCrrrr=rEr#r#r8r$r:s.  rc szeZdZdZd eeeeeeeffeeeeefefeeeeeeffddfdd Z e j e j d d d Z Z S) BeitConvModuleaD A convolutional block that bundles conv/norm/activation layers. This block simplifies the usage of convolution layers, which are commonly used with a norm layer (e.g., BatchNorm) and activation layer (e.g., ReLU). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rFrN) in_channels out_channelsrepaddingr{dilationr)cs<ttj||||||d|_t||_t|_dS)N)r r rer r{r ) r5r6rrnconv BatchNorm2dbnZReLU activation)r7r r rer r{r r8r#r$r6s  zBeitConvModule.__init__r&r)cCs"||}||}||}|Sr4)r rr)r7r&r1r#r#r$r=s   zBeitConvModule.forward)rFr)rr r!r"rrrrDrr6r/rCr=rEr#r#r8r$rs rcs<eZdZeeeddfdd ZejejdddZZS)BeitPyramidPoolingBlockN) pool_scaler channelsr)csLtt|t||ddg|_t|jD]\}}|t||q.dS)Nrre) r5r6rZAdaptiveAvgPool2drlayersr add_modulerD)r7rr rrrr8r#r$r6s   z BeitPyramidPoolingBlock.__init__rcCs|}|jD] }||}q |Sr4)r)r7r&Z hidden_staterr#r#r$r=s  zBeitPyramidPoolingBlock.forward) rr r!rr6r/rCr=rEr#r#r8r$rs rcsNeZdZdZeedfeeeddfdd Zej e ej ddd Z Z S) BeitPyramidPoolingModulea Pyramid Pooling Module (PPM) used in PSPNet. Args: pool_scales (tuple[int]): Pooling scales used in Pooling Pyramid Module. in_channels (int): Input channels. channels (int): Channels after modules, before conv_seg. align_corners (bool): align_corners argument of F.interpolate. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. .N) pool_scalesr r align_cornersr)csht||_||_||_||_g|_t|D]2\}}t|||d}|j || t ||q0dS)N)rr r) r5r6rrr rblocksrrappendrrD)r7rr rrrrblockr8r#r$r6s  z!BeitPyramidPoolingModule.__init__)rr)cCsHg}|jD]8}||}tjj||ddd|jd}||q |S)Nrqbilinearr[moder)rrr interpolater[rr)r7rZppm_outsppmZppm_outZupsampled_ppm_outr#r#r$r=s  z BeitPyramidPoolingModule.forward) rr r!r"rrrr6r/rCrr=rEr#r#r8r$rs " rcsDeZdZdZeddfdd ZddZejejdd d Z Z S) BeitUperHeadz Unified Perceptual Parsing for Scene Understanding. This head is the implementation of [UPerNet](https://arxiv.org/abs/1807.10221). Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. NrGcst|j|_|jgd|_|j|_d|_tj|j|j dd|_ t |j|jd|j|jd|_ t |jdt|j|j|jddd|_t|_t|_|jddD]@}t ||jdd}t |j|jddd}|j||j|qt t|j|j|jddd|_dS) NFrrrX)rr rer )r5r6rrKr rrrrnrrr psp_modulesrr bottleneckr lateral_convs fpn_convsrfpn_bottleneck)r7rHr Zl_convZfpn_convr8r#r$r6s>    zBeitUperHead.__init__cCs:|d}|g}|||tj|dd}||}|SrW)extendr&r/r^r')r7rrZpsp_outsr1r#r#r$ psp_forwards  zBeitUperHead.psp_forwardencoder_hidden_statesr)csfddtjDt}t|dddD]H}|djdd}|dtjj ||dj d|d<q@fd dt|dD}|dt|dddD]0}tjj |||djdddj d||<qt j |dd } |}|}|S) Ncsg|]\}}||qSr#r#)rrZ lateral_conv)r.r#r$r'sz(BeitUperHead.forward..rrrXrqrrcsg|]}j||qSr#)r)r)lateralsr7r#r$r4srY)rr(rr,rrr-rrr!rr/r^r*r)r7r.Zused_backbone_levelsrZ prev_shapeZfpn_outsr1r#)r.r/r7r$r=%s0   zBeitUperHead.forward) rr r!r"rr6r,r/rCr=rEr#r#r8r$r#s& r#c sTeZdZdZd eeeeeeeeffddfdd Ze j e j d d d Z Z S) BeitFCNHeada Fully Convolution Networks for Semantic Segmentation. This head is implemented of [FCNNet](https://arxiv.org/abs/1411.4038>). Args: config (BeitConfig): Configuration. in_channels kernel_size (int): The kernel size for convs in the head. Default: 3. dilation (int): The dilation rate for convs in the head. Default: 1. Based on OpenMMLab's implementation, found in https://github.com/open-mmlab/mmsegmentation. rqr rN)rHin_indexrer r)c st|j|_|j|_|j|_|j|_ ||_ |d|}g}| t |j|j|||dt |jdD] }| t |j|j|||dqj|jdkrt|_n tj||_|j rt |j|j|j||dd|_tj|j|jdd|_dS)Nrq)rer r rrr%r)r5r6rKr Zauxiliary_channelsrZauxiliary_num_convsZ num_convsZauxiliary_concat_input concat_inputr1rrrrrconvs Sequentialconv_catrnrr)r7rHr1rer Z conv_paddingr3rr8r#r$r6RsL      zBeitFCNHead.__init__r-cCs@||j}||}|jr2|tj||gdd}||}|S)NrrY)r1r3r2r5r/r^r)r7r.r;r1r#r#r$r=ts    zBeitFCNHead.forward)rqr r) rr r!r"rrrrr6r/rCr=rEr#r#r8r$r0Cs"r0zf Beit Model transformer with a semantic segmentation head on top e.g. for ADE20k, CityScapes. c seZdZeddfdd ZddZeeee e dd e e j e e j e e j e ee ee eeee fdd d ZZS) BeitForSemanticSegmentationNrGc st||j|_t|dd|_ttj|j|jdddt |jt tj|j|jddd|_ ttj|j|jddd|_ t |_tjddd|_t||_|jrt|nd|_|dS)NFrrqrd)r5r6rrrrr4rrKrZGELUfpn1fpn2rfpn3Z MaxPool2dfpn4r# decode_headZuse_auxiliary_headr0auxiliary_headrrr8r#r$r6s"    z$BeitForSemanticSegmentation.__init__c Cstjj||jddddd}|dk rDtjj||jddddd}t|jjd}|||}|}|dk r|||} ||jj| 7}|S)NrrFr)Z ignore_index)rrr!r-r rHZsemantic_loss_ignore_indexZauxiliary_loss_weight) r7rauxiliary_logitsrZupsampled_logitsZupsampled_auxiliary_logitsrZ main_lossrZauxiliary_lossr#r#r$ compute_losss(    z(BeitForSemanticSegmentation.compute_lossrrcs|dk r |njj}|dk r |njj}j|||d|d}|rF|jn|d}fddt|D} |jdjjjjfdd| D} j j j j g} t t| D]} | | | | | | <q| } d} jdk r| } d}|dk r"jjdkrtd n| | |}|sl|rB| f|dd}n| f|d d}|dk rh|f|S|St|| |r~|jnd|jd S) aV labels (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Ground truth semantic segmentation maps for computing the loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1`, a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoImageProcessor, BeitForSemanticSegmentation >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> model = BeitForSemanticSegmentation.from_pretrained("microsoft/beit-base-finetuned-ade-640-640") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # logits are of shape (batch_size, num_labels, height, width) >>> logits = outputs.logits ```NTrrcs$g|]\}}|djjkr|qSr*)rHZ out_indices)ridxfeaturer@r#r$rsz7BeitForSemanticSegmentation.forward..rcs<g|]4}|ddddddfddddqS)NrrrqrX)rZreshaper)r`patch_resolutionr#r$rsz/The number of labels should be greater than onerqr)rHrrrr;rr-rfrgr7r8r9r:rrr;r<rrrr>rr)r7rUrrrrrrr.featuresZopsrrr=rr1r#)r`rAr7r$r=sP$       z#BeitForSemanticSegmentation.forward)NNNNNN)rr r!rr6r>rrrrrrr/rCrrrr=rEr#r#r8r$r6~s&  r6)r%F)Or"collections.abcrjr dataclassesrtypingrrrrr/Ztorch.utils.checkpointrZtorch.nnrr r Z activationsr Zmodeling_outputsr rrrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrrZconfiguration_beitrZ get_loggerrloggerrrrrrZ"BEIT_PRETRAINED_MODEL_ARCHIVE_LISTrrCrBrr2Moduler3rFrNrwrrrrrrrrZBEIT_START_DOCSTRINGrrrrrrrrr#r0r6r#r#r#r$s       )'M( =&L  Y]R%%U;