U ,-e@sdZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!m"Z"m#Z#d dl$m%Z%e"&e'Z(dZ)dZ*dgZ+eGdddeZ,Gddde j-Z.Gddde j-Z/Gddde j-Z0Gddde j-Z1Gddde j-Z2Gddde j-Z3Gdd d e j-Z4Gd!d"d"e j-Z5Gd#d$d$e j-Z6Gd%d&d&e j-Z7Gd'd(d(eZ8d)Z9d*Z:d+Z;e d,e9Gd-d.d.e8ZGd4d5d5e j-Z?Gd6d7d7e j-Z@e d8e9Gd9d:d:e8ZAe d;e9Gde;Gd?d@d@e8ZCe dAe9GdBdCdCe8ZDdS)Dz PyTorch ViLT model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) ViltConfigrzdandelin/vilt-b32-mlmc@sfeZdZUdZdZeejed<dZ ejed<dZ ee e ejed<dZ ee e ejed<dS)(ViltForImagesAndTextClassificationOutputa Class for outputs of [`ViltForImagesAndTextClassification`]. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`List[tuple(torch.FloatTensor)]`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): List of tuples of `torch.FloatTensor` (one for each image-text pair, each tuple containing the attention weights of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlosslogits hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrrr'r'g/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/vilt/modeling_vilt.pyr9s rcs4eZdZdZfddZd ddZd dd ZZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. cstt||_ttdd|j|_ t ||_ |j j }ttd|d|j|_ t|j|j|_t|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterr$zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings EmbeddingZmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfr<r5 __class__r'r(r,\s   zViltEmbeddings.__init__cs|jjjj\}}}}||}|dddddddf}tjj||jd|jdfd}|dddfj dddddf} |dddfj dddddf} |j\} } |j j |j j } |j ddddddfddd| | | tjfddt| | Ddd}|ddd}|ddd}tjtt|jd t|jd d d d dj|jd }|ddddddddf}||jd|jdd d d }|dd}|d}dksdkstts| | }|n| | }t||jddd|jdddddf}fdd|D}fdd|D}dd|D}dd|D}fdd|D}g}t t|||D]\}\}}}|dkrt!t"|}|#|||n>tj!t"||dd}|#tj|||||gddqtj|dd}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }tj||fdd}tj|j dddddfdddddf| d d |fdd}||}|%|}tjt"|jdd||gdd}|||fffS)Nr sizerrdimc sBg|]:\}}tjtjj||fdddd|d|fqS)ZbilinearT)rCmodeZ align_cornersr)r functionalpad interpolate).0hw)height spatial_poswidthr'r( xs z/ViltEmbeddings.visual_embed..Zij)ZindexingdeviceF)as_tuplecs$g|]}dddf|kqSNrr'rJu) valid_idxr'r(rPscs$g|]}dddf|kqSrVr'rW) non_valid_idxr'r(rPscSsg|]}|dqSrrBrJvr'r'r(rPscSsg|]}|dqSr[rBr\r'r'r(rPscsg|] }|qSr'r'r\max_image_lengthr'r(rPsT) replacement)&r4 projectionweightshapefloatrrGrIlongsumr< image_size patch_sizer6 transposeviewr$catzipflattenstackrarangetorTexpand isinstanceintmaxminZnonzerounique enumerateZ multinomialonesappendr2r;)r= pixel_values pixel_maskr__phpwxZx_maskZx_hZx_w batch_size num_channelsZ patch_dimZ pos_embed patch_indexZeffective_resolutionZ unique_rowsZ valid_row_idxZnon_valid_row_idxZ valid_numsZnon_valid_numsZpad_numsselectir]nvpZ valid_choiceZ pad_choiceZ cls_tokensr')rMr_rZrNrYrOr( visual_embedks|  $$$0  $      (.,..6 &zViltEmbeddings.visual_embedrc  Cs|j|||d} |dkr4|j|||jjd\}} } n |d} |dkrJd}| |tj|tj| j d} ||tj | |tj| j d}tj | |gdd} tj || gdd} | | fS)N) input_idstoken_type_ids inputs_embedsr^rdtyperTrD) r.rr<r_rmr8r$Z zeros_likererTZ full_likerk)r=rattention_maskrrzr{r image_embedsimage_token_type_idxZ text_embedsZ image_masksr embeddingsmasksr'r'r(forwards.  zViltEmbeddings.forward)r@)r)r r!r"r#r,rr __classcell__r'r'r>r(r)Ss   ar)cs*eZdZdZfddZdddZZS)r-zGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxZepsposition_embedding_typeabsolute position_ids)rrRF) persistentr)r)r+r,rr7 vocab_sizer1Z pad_token_idword_embeddingsZmax_position_embeddingsr6Ztype_vocab_sizer8 LayerNormlayer_norm_epsr9r:r;getattrrZregister_bufferr$rorqr0rrCrer=r<r>r'r(r,s" zTextEmbeddings.__init__Nc Cs|dk r|}n|dd}|d}|dkrH|jddd|f}|dkrt|dr|jddd|f}||d|}|}ntj|tj|jjd}|dkr| |}| |} || } |j dkr| |} | | 7} | | } || } | S)NrRrrrrr)rCrhasattrrrqr$r0rerTrr8rr6rr;) r=rrrr input_shape seq_lengthZbuffered_token_type_idsZ buffered_token_type_ids_expandedr8rr6r'r'r(rs,        zTextEmbeddings.forward)NNNNr r!r"r#r,rrr'r'r>r(r-s r-cs(eZdZdZfddZddZZS)r3z# Image to Patch Embedding. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ t j ||||d|_ dS)Nrr)Z kernel_sizeZstride)r+r,rgrhrr1rr collectionsabcIterabler5rConv2dra)r=r<rgrhrr1r5r>r'r(r,+s  zViltPatchEmbeddings.__init__cCs.|j\}}}}||jkr td||}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)rcr ValueErrorra)r=rzrrrMrOrr'r'r(r:s  zViltPatchEmbeddings.forwardrr'r'r>r(r3&s r3cs.eZdZfddZddZd ddZZS) ViltSelfAttentioncst|j|jdkr@t|ds@td|jfd|jd|j|_t|j|j|_|j|j|_t j |j|j|j d|_ t j |j|j|j d|_ t j |j|j|j d|_t |j|_dS)NrZembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r+r,r1num_attention_headsrrrsattention_head_size all_head_sizerLinearZqkv_biasquerykeyvaluer9Zattention_probs_dropout_probr;rr>r'r(r,Es zViltSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)NrRrrArr )rCrrrjpermute)r=rZ new_x_shaper'r'r(transpose_for_scoresWs z&ViltSelfAttention.transpose_for_scoresNFcCs||}|||}|||}||}t||dd} | t|j } |dk rh| |} t j dd| } | | } |dk r| |} t| |} | dddd} | dd|jf} | j| } |r| | fn| f} | S)NrRrQrDrrArr )rrrrr$matmulrimathsqrtrrZSoftmaxr;r contiguousrCrrj)r=rr head_maskoutput_attentionsZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr'r'r(r\s$     zViltSelfAttention.forward)NNF)r r!r"r,rrrr'r'r>r(rDs rcs@eZdZdZeddfdd ZejejejdddZZ S) ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. Nr<returncs.tt|j|j|_t|j|_dSN) r+r,rrr1denser9r:r;rr>r'r(r,s zViltSelfOutput.__init__r input_tensorrcCs||}||}|Srrr;r=rrr'r'r(rs  zViltSelfOutput.forward) r r!r"r#rr,r$Tensorrrr'r'r>r(rsrcs.eZdZfddZddZd ddZZS) ViltAttentioncs*tt||_t||_t|_dSr)r+r,r attentionroutputset pruned_headsrr>r'r(r,s   zViltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrD)lenrrrrrrrrrrrrunion)r=headsindexr'r'r( prune_headss zViltAttention.prune_headsNFcCs6|||||}||d|}|f|dd}|S)Nrr)rr)r=rrrrZ self_outputsattention_outputrr'r'r(rszViltAttention.forward)NNF)r r!r"r,rrrr'r'r>r(rs rcs8eZdZeddfdd ZejejdddZZS)ViltIntermediateNrcsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r+r,rrr1intermediate_sizerrr hidden_actstrr intermediate_act_fnrr>r'r(r,s   zViltIntermediate.__init__)rrcCs||}||}|Sr)rrr=rr'r'r(rs  zViltIntermediate.forward r r!r"rr,r$rrrr'r'r>r(rsrcs<eZdZeddfdd ZejejejdddZZS) ViltOutputNrcs.tt|j|j|_t|j|_ dSr) r+r,rrrr1rr9r:r;rr>r'r(r,s zViltOutput.__init__rcCs ||}||}||}|Srrrr'r'r(rs  zViltOutput.forwardrr'r'r>r(rsrcs*eZdZdZfddZdddZZS) ViltLayerz?This corresponds to the Block class in the timm implementation.csbt|j|_d|_t||_t||_t||_ t j |j |j d|_t j |j |j d|_dS)Nrr)r+r,Zchunk_size_feed_forwardZ seq_len_dimrrr intermediaterrrrr1rlayernorm_beforelayernorm_afterrr>r'r(r,s    zViltLayer.__init__NFc Csj|j|||||d}|d}|dd}|||j}||}||}|||}|f|}|S)Nrrr)rrrprTrrr) r=rrrrZself_attention_outputsrrZ layer_outputr'r'r(rs     zViltLayer.forward)NNFrr'r'r>r(rs rcs&eZdZfddZdddZZS) ViltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr')r)rJr|r<r'r(rPsz(ViltEncoder.__init__..F) r+r,r<rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingrr>rr(r,s  zViltEncoder.__init__NFTcs|rdnd}rdnd}t|jD]\} } |r8||f}|dk rH|| nd} |jr~|jr~fdd} tjj| | ||| } n| ||| } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr'csfdd}|S)Ncs|fSrr')inputs)modulerr'r(custom_forwardszJViltEncoder.forward..create_custom_forward..custom_forwardr')rrr)rr(create_custom_forwardsz2ViltEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSrr'r\r'r'r( /sz&ViltEncoder.forward..)last_hidden_staterr) rwrrZtrainingr$utils checkpointtupler )r=rrrroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsrZ layer_moduleZlayer_head_maskrZ layer_outputsr'rr(rs6      zViltEncoder.forward)NNFFTr r!r"r,rrr'r'r>r(rs rc@s6eZdZdZeZdZdZddgZddZ d d d Z d S) ViltPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. viltTr)rcCst|tjtjfr@|jjjd|jjd|j dk r|j j nft|tj r|jjjd|jjd|j dk r|jj|j  n&t|tj r|j j |jjddS)zInitialize the weightsg)ZmeanZstdNg?)rrrrrrbdataZnormal_r<Zinitializer_rangerZzero_r7rrZfill_)r=rr'r'r( _init_weightsBs     z!ViltPreTrainedModel._init_weightsFcCst|tr||_dSr)rrrr)r=rrr'r'r(_set_gradient_checkpointingRs z/ViltPreTrainedModel._set_gradient_checkpointingN)F) r r!r"r#r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrrr'r'r'r(r7sraH This model is a PyTorch `torch.nn.Module `_ subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`ViltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) pixel_values (`torch.FloatTensor` of shape `(batch_size, num_images, num_channels, height, width)`): Pixel values. Pixel values can be obtained using [`AutoImageProcessor`]. See [`ViltImageProcessor.__call__`] for details. pixel_mask (`torch.LongTensor` of shape `(batch_size, num_images, height, width)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: - 1 for pixels that are real (i.e. **not masked**), - 0 for pixels that are padding (i.e. **masked**). `What are attention masks? <../glossary.html#attention-mask>`__ head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. image_embeds (`torch.FloatTensor` of shape `(batch_size, num_images, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare ViLT Model transformer outputting raw hidden-states without any specific head on top.cseZdZdfdd ZddZddZdd Zeee e e d de e je e je e je e je e je e je e je e je ee ee ee eee ee jfd d dZZS) ViltModelTcsXt|||_t||_t||_tj|j |j d|_ |rFt |nd|_ |dSNr)r+r,r<r)rrencoderrrr1r layernorm ViltPoolerpooler post_init)r=r<add_pooling_layerr>r'r(r,s   zViltModel.__init__cCs |jjjSrrr.rr=r'r'r(get_input_embeddingsszViltModel.get_input_embeddingscCs||jj_dSrr)r=rr'r'r(set_input_embeddingsszViltModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r=Zheads_to_prunerrr'r'r( _prune_headsszViltModel._prune_heads output_typerN) rrrrzr{rrrrrrrrc  Cs| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd| \}}|dk r|jn|j}|dkrtj ||f|d}|dk r|dk rtdn|dkr|dkrtd|dk r|j dn|j d}||kr.td |dkrTtj ||jj |jj f|d}| ||jj }|j|||||||| d \}}||| }|j|||| | | d }|d}||}|jdk r||nd}| s||f|d dSt|||j|jd S)a Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerRz5You have to specify either input_ids or inputs_embedsrSzFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrrr)r pooler_outputrr)r<rruse_return_dictrZ%warn_if_padding_and_no_attention_maskrCrTr$rxrcrgZ get_head_maskrrZget_extended_attention_maskrrrr rr)r=rrrrzr{rrrrrrrrZtext_batch_sizerrTZimage_batch_sizeZembedding_outputZextended_attention_maskZencoder_outputssequence_output pooled_outputr'r'r(rsp'        zViltModel.forward)T) NNNNNNNNNNNN)r r!r"r,rrrrVILT_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr$ LongTensorr%rsboolrrrrr'r'r>r(rsB  rcs$eZdZfddZddZZS)rcs*tt|j|j|_t|_dSr)r+r,rrr1rZTanh activationrr>r'r(r,gs zViltPooler.__init__cCs(|dddf}||}||}|SrV)rr)r=rZfirst_token_tensorr r'r'r(rls  zViltPooler.forwardrr'r'r>r(rfs rzU ViLT Model with a language modeling head on top as done during pretraining. cseZdZddgZfddZddZddZee d e e e d de eje eje eje eje eje eje eje eje eje ee ee eee eejfd d dZZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biascs,t|t||_t||_|dSr)r+r,rr ViltMLMHead mlm_scorerrr>r'r(r,~s   zViltForMaskedLM.__init__cCs|jjSrrdecoderrr'r'r(get_output_embeddingssz%ViltForMaskedLM.get_output_embeddingscCs ||j_dSrr)r=Znew_embeddingsr'r'r(set_output_embeddingssz%ViltForMaskedLM.set_output_embeddingszbatch_size, sequence_lengthrN rrrrzr{rrrlabelsrrrrc  Cs| dk r | n|jj} |j||||||||| | | d } | dd\}}|dk rV|jdn|jd}|ddd|f|dd|df}}||}d}| dk rt}| |j} ||d|jj | d}| s|f| dd}|dk r|f|S|St ||| j | j dS)a labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrzr{rrrrrrrArrRrrrr) r<rrrcrrrprTrjrr rr)r=rrrrzr{rrrrrrrrr r Z text_seq_lenZ text_featuresr|Z mlm_logitsZmasked_lm_lossloss_fctrr'r'r(rs@I*  zViltForMaskedLM.forward) NNNNNNNNNNNN)r r!r"Z_tied_weights_keysr,rrrr formatrr r rr$r r%rrrrrr'r'r>r(rusB   rcs$eZdZfddZddZZS)ViltPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dSr) r+r,rrr1rrrrrr transform_act_fnrrrr>r'r(r,s   z$ViltPredictionHeadTransform.__init__cCs"||}||}||}|Sr)rrrrr'r'r(r s   z#ViltPredictionHeadTransform.forwardrr'r'r>r(rs rcs&eZdZdfdd ZddZZS)rNcsbt||_t||_tj|j|jdd|_ t t |j|_ |dk rT||j _|j |j _ dS)NFr)r+r,r<r transformrrr1rrr/r$r0rrb)r=r<rbr>r'r(r,s  zViltMLMHead.__init__cCs||}||}|Sr)rr)r=rr'r'r(rs  zViltMLMHead.forward)Nrr'r'r>r(rs rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) ViltForQuestionAnsweringc slt||j|_t||_tt|j|jdt |jdt t|jd|j|_ | dS)NrA) r+r, num_labelsrrr Sequentialrr1rGELU classifierrrr>r'r(r,,s  z!ViltForQuestionAnswering.__init__rNrc  Cs| dk r | n|jj} |j||||||||| | | d } | r>| jn| d}||}d}| dk r| |j} tj || | j d}| s|f| dd}|dk r|f|S|St ||| j | j dS)a labels (`torch.FloatTensor` of shape `(batch_size, num_labels)`, *optional*): Labels for computing the visual question answering loss. This tensor must be either a one-hot encoding of all answers that are applicable for a given example in the batch, or a soft encoding indicating which answers are applicable, where 1.0 is the highest score. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrrrAr)r<rrrr$rprTrrGZ binary_cross_entropy_with_logitsrcrrrr=rrrrzr{rrrrrrrrrrrrr'r'r(r=s:1  z ViltForQuestionAnswering.forward) NNNNNNNNNNNNr r!r"r,rr rrr rr$r r%rrrrrr'r'r>r(r $s<  r z Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) ViltForImageAndTextRetrievalcs2t|t||_t|jd|_|dSr*) r+r,rrrrr1 rank_outputrrr>r'r(r,s  z%ViltForImageAndTextRetrieval.__init__rNrc  Cs| dk r | n|jj} |j||||||||| | | d } | r>| jn| d}||}d}| dk rp| |j} td| s|f| dd}|dk r|f|S|St||| j | j dS)a' labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NrrzTraining is not yet supported.rAr) r<rrrr(rprTNotImplementedErrorrrrr%r'r'r(rs:-  z$ViltForImageAndTextRetrieval.forward) NNNNNNNNNNNNr&r'r'r>r(r's<  r'zq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) "ViltForImagesAndTextClassificationc svt||j|_t||_|j}tt|j ||j |t |j |t t|j ||j|_ | dSr)r+r,r!rr num_imagesrr"rr1rr#r$r)r=r<r+r>r'r(r,s  z+ViltForImagesAndTextClassification.__init__rNrc CsN| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |dk rX|jdkrX|d}|dk rt|jdkrt|d}|dk r|jdnd} | dkr|dk r|jdnd} | |jjkrtdg}| rgnd}| rgnd}t | D]}|j ||||dk r|dd|ddddddfnd|dk rB|dd|ddddfnd|||dk rn|dd|ddddfnd|d| | | d }| r|j n|d}| || r| |j | r| |jqtj|dd}||}d}| dk rt}| |j} ||d|j| d}| s>|||f}|dk r:|f|S|St||||d S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Returns: Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrr z\Make sure to match the number of images in the model with the number of images in the input.) rrrzr{rrrrrrrrRrDr)r<rrrndimZ unsqueezercr+rrrrryrrr$rkr$rrprTrjr!r)r=rrrrzr{rrrrrrrr+Zpooler_outputsrrrrrr rrrrr'r'r(rsn/      .((      z*ViltForImagesAndTextClassification.forward) NNNNNNNNNNNN)r r!r"r,rr rrr rr$r r%rrrrrr'r'r>r(r*s<  r*z ViLT Model with a token classification head on top (a linear layer on top of the final hidden-states of the text tokens) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeeeeedde e j e e j e e j e e j e e j e e j e e j e e j e e j e e e e e e eeee j fd ddZZS) ViltForTokenClassificationcsNt||j|_t|dd|_t|j|_t |j |j|_ | dS)NF)r) r+r,r!rrrr9r:r;rr1r$rrr>r'r(r,s  z#ViltForTokenClassification.__init__rNrc  Cs| dk r | n|jj} |j||||||||| | | d } | d}|dk rN|jdn|jd}||}||ddd|f}d}| dk rt}| |j} || d|j | d}| s|f| dd}|dk r|f|S|St ||| j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: NrrrrRrAr)r<rrrcr;r$rrprTrjr!rrr)r=rrrrzr{rrrrrrrrr Ztext_input_sizerrrrr'r'r(rs@  z"ViltForTokenClassification.forward) NNNNNNNNNNNN)r r!r"r,rr rrr rr$r r%rrrrrr'r'r>r(r.s<  r.)Er#collections.abcrr dataclassesrtypingrrrrr$Ztorch.utils.checkpointrZtorch.nnrZ activationsr Zmodeling_outputsr r r rrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_viltrZ get_loggerr loggerr Z_CHECKPOINT_FOR_DOCZ"VILT_PRETRAINED_MODEL_ARCHIVE_LISTrModuler)r-r3rrrrrrrrZVILT_START_DOCSTRINGr Z4VILT_IMAGES_AND_TEXT_CLASSIFICATION_INPUTS_DOCSTRINGrrrrrr r'r*r.r'r'r'r(s        9=#&;  78j_