U 9%e@sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZmZd d lm Z m!Z!m"Z"e#e$Z%dZ&dZ'dgZ(dZ)dZ*dZ+dZ,eGdddeZ-eGdddeZ.eGdddeZ/e j0e j0dddZ1e j0e j0dddZ2e"e3d d!d"Z4dSee3efe5d$d%d&Z6Gd'd(d(e j7Z8Gd)d*d*e j9Z:Gd+d,d,e j7Z;Gd-d.d.e j7ZGd3d4d4e j7Z?Gd5d6d6e j7Z@Gd7d8d8e j7ZAGd9d:d:e j7ZBGd;d<dd>e j7ZDGd?d@d@e j7ZEGdAdBdBe j7ZFGdCdDdDe j7ZGGdEdFdFe j7ZHGdGdHdHe j7ZIGdIdJdJeZJedKe)GdLdMdMeJZKedNe)GdOdPdPeJZLee)GdQdRdReJZMdS)Tz PyTorch ALIGN model.N) dataclass)AnyOptionalTupleUnion)nn)ACT2FN)BaseModelOutputWithNoAttention)BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions(BaseModelOutputWithPoolingAndNoAttention)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) AlignConfigAlignTextConfigAlignVisionConfigzkakaobrain/align-basera> This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`AlignConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`EfficientNetImageProcessor.__call__`] for details. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. aW Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using [`AutoImageProcessor`]. See [`EfficientNetImageProcessor.__call__`] for details. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. c@sHeZdZUdZdZeejed<dZ ejed<dZ ee ejed<dS)AlignVisionModelOutputa+ Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. Args: image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. N image_embedslast_hidden_state hidden_states) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrr&r&g/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/align/modeling_align.pyrs rc@s^eZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)AlignTextModelOutputa Base class for text model's outputs that also contains a pooling of the last hidden states. Args: text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. N text_embedsrr attentions) rr r!r"r)rr#r$r%rrrr*r&r&r&r'r(s r(c@seZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ejed<dZ ejed<dZ eed<dZeed <eed d d ZdS) AlignOutputa Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AlignTextModel`]. image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`): The output of [`AlignVisionModel`]. text_model_output(`BaseModelOutputWithPoolingAndCrossAttentions`): The output of the [`AlignTextModel`]. vision_model_output(`BaseModelOutputWithPoolingAndNoAttention`): The output of the [`AlignVisionModel`]. Nlosslogits_per_imagelogits_per_textr)rtext_model_outputvision_model_outputreturncstfddDS)Nc3s,|]$}|dkr|n t|VqdS))r/r0N)getattrto_tuple).0kselfr&r' sz'AlignOutput.to_tuple..)tuplekeysr7r&r7r'r4s zAlignOutput.to_tuple)rr r!r"r,rr#r$r%r-r.r)rr/r r0r rrr4r&r&r&r'r+s   r+)logitsr2cCs"tjj|tjt||jdddS)Ndeviceg?)Zlabel_smoothing)r functionalZ cross_entropyr#arangelenr>)r<r&r&r'contrastive_losssrB) similarityr2cCs t|}t|}||dS)Ng@)rBt)rCZ caption_lossZ image_lossr&r&r' align_losss rE)config num_channelscCsJ|j}||j9}t|t||d||}|d|krB||7}t|S)z< Round number of filters based on depth multiplier. g?)Z depth_divisorZwidth_coefficientmaxint)rFrGZdivisorZnew_dimr&r&r' round_filters"s   rKT) kernel_sizeadjustcCsrt|tr||f}|dd|ddf}|rR|dd|d|dd|dfS|d|d|d|dfSdS)aJ Utility function to get the tuple padding value for the depthwise convolution. Args: kernel_size (`int` or `tuple`): Kernel size of the convolution layers. adjust (`bool`, *optional*, defaults to `True`): Adjusts padding value to apply to right and bottom sides of the input. rrHrN) isinstancerJ)rLrMZcorrectr&r&r' correct_pad2s $rOcs:eZdZdZedfdd ZejejdddZZ S)AlignVisionEmbeddingszL A module that corresponds to the stem module of the original work. rFcshtt|d|_tjdd|_tj|j|jddddd|_ tj |j|j |j d |_ t|j|_dS) N )rrrrpaddingrrHvalidFrLstriderTbias)epsmomentum)super__init__rKout_dimr ZeroPad2drTConv2drG convolution BatchNorm2dbatch_norm_epsbatch_norm_momentum batchnormr hidden_act activationr8rF __class__r&r'r\Ls  zAlignVisionEmbeddings.__init__) pixel_valuesr2cCs,||}||}||}||}|SN)rTr`rdrf)r8rjfeaturesr&r&r'forwardWs     zAlignVisionEmbeddings.forward) rr r!r"rr\r#Tensorrm __classcell__r&r&rhr'rPGs rPcseZdZdfdd ZZS) AlignVisionDepthwiseConv2drrrTzerosc s*||} tj|| |||||||d dS)N) in_channels out_channelsrLrWrTdilationgroupsrX padding_mode)r[r\) r8rrZdepth_multiplierrLrWrTrtrXrvrsrhr&r'r\bs z#AlignVisionDepthwiseConv2d.__init__)rrrrrTrq)rr r!r\ror&r&rhr'rpasrpcs@eZdZdZeeeedfdd Zejej dddZ Z S)AlignVisionExpansionLayerz_ This corresponds to the expansion phase of each block in the original implementation. rFin_dimr]rWcsBttj||dddd|_tj||jd|_t|j |_ dS)NrsameFrrrsrLrTrX) num_featuresrY) r[r\rr_ expand_convrarb expand_bnr re expand_act)r8rFryr]rWrhr&r'r\s z"AlignVisionExpansionLayer.__init__rr2cCs"||}||}||}|Srk)r}r~rr8rr&r&r'rms   z!AlignVisionExpansionLayer.forward) rr r!r"rrJr\r#r$rnrmror&r&rhr'rw|s rwcsBeZdZdZeeeeedfdd Zej ej dddZ Z S)AlignVisionDepthwiseLayerzk This corresponds to the depthwise convolution phase of each block in the original implementation. rFryrWrLadjust_paddingcsvt||_|jdkrdnd}t||d}tj|d|_t||||dd|_tj ||j |j d|_ t |j|_dS) NrHrUrz)rMrSFrVr|rYrZ)r[r\rWrOrr^depthwise_conv_padrpdepthwise_convrarbrcdepthwise_normr re depthwise_act)r8rFryrWrLrZconv_padrTrhr&r'r\s$  z"AlignVisionDepthwiseLayer.__init__rcCs6|jdkr||}||}||}||}|S)NrH)rWrrrrrr&r&r'rms      z!AlignVisionDepthwiseLayer.forward rr r!r"rrJboolr\r#r$rnrmror&r&rhr'rsrcsBeZdZdZd eeeedfdd Zej ej dddZ Z S) AlignVisionSqueezeExciteLayerzl This corresponds to the Squeeze and Excitement phase of each block in the original implementation. FrFry expand_dimexpandcst|r|n||_tdt||j|_tjdd|_ tj |j|jddd|_ tj |j|jddd|_ t |j|_t|_dS)Nr)Z output_sizerz)rrrsrLrT)r[r\dimrIrJZsqueeze_expansion_ratioZdim_serZAdaptiveAvgPool2dsqueezer_reducerr re act_reduceZSigmoid act_expand)r8rFryrrrhr&r'r\s$  z&AlignVisionSqueezeExciteLayer.__init__rcCsF|}||}||}||}||}||}t||}|Srk)rrrrrr#mul)r8rinputsr&r&r'rms      z%AlignVisionSqueezeExciteLayer.forward)Frr&r&rhr'rsrcsHeZdZdZeeeeeedfdd Ze j e j e j dddZ Z S)AlignVisionFinalBlockLayerz[ This corresponds to the final phase of each block in the original implementation. rFryr]rW drop_rateid_skipcsXt|dko| |_tj||dddd|_tj||j|jd|_ tj |d|_ dS)NrrzFr{r)p) r[r\ apply_dropoutrr_ project_convrarbrc project_bnDropoutdropout)r8rFryr]rWrrrhr&r'r\s z#AlignVisionFinalBlockLayer.__init__) embeddingsrr2cCs0||}||}|jr,||}||}|Srk)rrrr)r8rrr&r&r'rms    z"AlignVisionFinalBlockLayer.forwardrr r!r"rrJfloatrr\r#r$rnrmror&r&rhr'rsrc sJeZdZdZeeeeeeeeed fdd Ze j e j dddZ Z S)AlignVisionBlocka This corresponds to the block module of original the EfficientNet vision encoder implementation. Args: config ([`AlignVisionConfig`]): Model configuration class. in_dim (`int`): Number of input channels. out_dim (`int`): Number of output channels. stride (`int`): Stride size to be used in convolution layers. expand_ratio (`int`): Expand ratio to set the output dimensions for the expansion and squeeze-excite layers. kernel_size (`int`): Kernel size for the depthwise convolution layer. drop_rate (`float`): Dropout rate to be used in the final phase of each block. id_skip (`bool`): Whether to apply dropout and sum the final hidden states with the input embeddings during the final phase of each block. Set to `True` for the first block of each stage. adjust_padding (`bool`): Whether to apply padding to only right and bottom side of the input kernel before the depthwise convolution operation, set to `True` for inputs with odd input sizes. ) rFryr]rW expand_ratiorLrrrc st||_|jdkrdnd|_||} |jrDt||| |d|_t||jrR| n|||| d|_t||| |jd|_ t ||jr| n|||||d|_ dS)NrTFrxrrr) r[r\rrrw expansionrrrsqueeze_exciter projection) r8rFryr]rWrrLrrrZ expand_in_dimrhr&r'r\"s@   zAlignVisionBlock.__init__rcCs<|}|jdkr||}||}||}|||}|S)Nr)rrrrr)r8rrr&r&r'rmKs     zAlignVisionBlock.forwardrr&r&rhr'rs)rcsFeZdZdZedfdd Zd ejee ee e ddd Z Z S) AlignVisionEncoderz Forward propogates the embeddings through each vision encoder (EfficientNet) block. Args: config ([`AlignVisionConfig`]): Model configuration class. rQc s8t|j_fddt|j}tfdd|jD}d}g}t|D]}t||j|}t||j |}|j |} |j |} |j |} t|j|D]} | dkrdnd} | dkrdn| } | dkr|n|}||j krdnd}|j||}t|||| | | || |d }|||d7}qqPt|_dS) Ncsttj|Srk)rJmathceildepth_coefficient)Zrepeatsr7r&r' round_repeatsesz2AlignVisionEncoder.__init__..round_repeatsc3s|]}|VqdSrkr&)r5n)rr&r'r9jsz.AlignVisionEncoder.__init__..rTFr) rFryr]rWrLrrrr)r[r\rrArrsumZnum_block_repeatsrangerKrsstridesZ kernel_sizesZ expand_ratiosZdepthwise_paddingZdrop_connect_raterappendr ModuleListblocks)r8rFZnum_base_blocksZ num_blocksZcurr_block_numriryr]rWrLrjrrrblockrh)rr8r'r\asB         zAlignVisionEncoder.__init__FT)routput_hidden_states return_dictr2cCsV|r |fnd}|jD]}||}|r||f7}q|sJtdd||fDSt||dS)Ncss|]}|dk r|VqdSrkr&r5vr&r&r'r9sz-AlignVisionEncoder.forward..)rr)rr:r )r8rrrall_hidden_statesrr&r&r'rms  zAlignVisionEncoder.forward)FT) rr r!r"rr\r#r$rrr rmror&r&rhr'rXs.rcsTeZdZdZfddZd eejeejeejeeje ej dddZ Z S) AlignTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_t|dd|_|jdt|jddd|jd tj|jtjd dddS) N) padding_idxrYposition_embedding_typeabsolute position_ids)rF) persistenttoken_type_idsdtype)r[r\r EmbeddingZ vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrhidden_dropout_probrr3rZregister_bufferr#r@rrqrsizelongrgrhr&r'r\s" zAlignTextEmbeddings.__init__Nr) input_idsrr inputs_embedspast_key_values_lengthr2c Cs|dk r|}n|dd}|d}|dkrL|jdd|||f}|dkrt|dr|jddd|f}||d|} | }ntj|tj|jjd}|dkr| |}| |} || } |j dkr| |} | | 7} | | } || } | S)Nrrrrrr>r)rrhasattrrrr#rqrr>rrrrrr) r8rrrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrrrr&r&r'rms,        zAlignTextEmbeddings.forward)NNNNr) rr r!r"r\rr# LongTensorr$rJrnrmror&r&rhr'rs rc seZdZd fdd ZejejdddZd ejeejeejeejeejee e ejee e ejdd d Z Z S) AlignTextSelfAttentionNcst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|pt|dd|_|jdks|jd kr|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rr relative_keyrelative_key_queryrHr)r[r\rnum_attention_headsr ValueErrorrJattention_head_size all_head_sizerLinearquerykeyvaluerZattention_probs_dropout_probrr3rrrdistance_embedding is_decoderr8rFrrhr&r'r\s* zAlignTextSelfAttention.__init__)xr2cCs6|dd|j|jf}||}|ddddS)NrrrHrr)rrrviewpermute)r8rZ new_x_shaper&r&r'transpose_for_scoress z+AlignTextSelfAttention.transpose_for_scoresFrattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsr2cCs||}|dk } | r4|dk r4|d} |d} |}n| r^|||} |||} |}nv|dk r|||} |||} tj|d| gdd} tj|d| gdd} n |||} |||} ||} |dk } |jr| | f}t| | dd}|j dks |j dkr| j d| j d}}| r^tj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n4|j dkrtd | |}td | |}|||}|t|j}|dk r:||}tjj|dd}||}|dk rf||}t|| }|dddd }|dd|jf}||}|r||fn|f}|jr||f}|S)NrrrHrrrrrrzbhld,lrd->bhlrzbhrd,lrd->bhlrr) rrrrr#catrmatmulZ transposershapetensorrr>rr@rrtorZeinsumrsqrtrrr?Zsoftmaxrr contiguousrr)r8rrrrrrrZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layer use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr&r&r'rmsp             zAlignTextSelfAttention.forward)N)NNNNNF) rr r!r\r#rnrrr$rrrmror&r&rhr'rs$rcs4eZdZfddZejejejdddZZS)AlignTextSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) r[r\rrrdenserrrrrrgrhr&r'r\ls zAlignTextSelfOutput.__init__r input_tensorr2cCs&||}||}|||}|Srkrrrr8rrr&r&r'rmrs  zAlignTextSelfOutput.forwardrr r!r\r#rnrmror&r&rhr'rks rc sveZdZd fdd ZddZd ejeejeejeejeejee e ejee e ejddd Z Z S) AlignTextAttentionNcs.tt||d|_t||_t|_dS)Nr)r[r\rr8routputset pruned_headsrrhr&r'r\{s  zAlignTextAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)rArr8rrrrrrrr rrunion)r8Zheadsindexr&r&r' prune_headss zAlignTextAttention.prune_headsFrc Cs<||||||||}||d|} | f|dd} | S)Nrr)r8r ) r8rrrrrrrZ self_outputsattention_outputrr&r&r'rms  zAlignTextAttention.forward)N)NNNNNF) rr r!r\rr#rnrr$rrrmror&r&rhr'r zs$r cs0eZdZfddZejejdddZZS)AlignTextIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSrk) r[r\rrrintermediate_sizerrNrestrr intermediate_act_fnrgrhr&r'r\s   zAlignTextIntermediate.__init__rcCs||}||}|Srk)rrrr&r&r'rms  zAlignTextIntermediate.forwardr r&r&rhr'rs rcs4eZdZfddZejejejdddZZS)AlignTextOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r[r\rrrrrrrrrrrgrhr&r'r\s zAlignTextOutput.__init__rcCs&||}||}|||}|Srkrrr&r&r'rms  zAlignTextOutput.forwardr r&r&rhr'rs rc steZdZfddZd ejeejeejeejeejeeeejee eejdddZ dd Z Z S) AlignTextLayercsrt|j|_d|_t||_|j|_|j|_|jrZ|jsLt|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrr )r[r\chunk_size_feed_forward seq_len_dimr  attentionradd_cross_attentionrcrossattentionr intermediaterr rgrhr&r'r\s   zAlignTextLayer.__init__NFrc Cs|dk r|ddnd}|j|||||d} | d} |jrP| dd} | d} n | dd} d} |jr|dk rt|dstd|d|dk r|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) NrHrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrrrrfeed_forward_chunkrr)r8rrrrrrrZself_attn_past_key_valueZself_attention_outputsrrZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputr&r&r'rmsV         zAlignTextLayer.forwardcCs||}|||}|Srk)rr )r8rZintermediate_outputr!r&r&r'r s  z!AlignTextLayer.feed_forward_chunk)NNNNNF) rr r!r\r#rnrr$rrrmr ror&r&rhr'rs$  ArcseZdZfddZd ejeejeejeejeejeeeejee ee ee ee e eeje fd ddZ Z S) AlignTextEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr&)r)r5_rQr&r' &sz-AlignTextEncoder.__init__..F) r[r\rFrrrnum_hidden_layerslayergradient_checkpointingrgrhrQr'r\#s  zAlignTextEncoder.__init__NFT) rrrrrpast_key_valuesrrrrr2c  st| rdnd} rdnd} r(|jjr(dnd} |jrJ|jrJ|rJtdd}|rRdnd}t|jD]\}}| rv| |f} |dk r||nd}|dk r||nd|jr|jrևfdd}tj j |||||||}n||||||}|d}|r||df7}r`| |df} |jjr`| |d f} q`| r@| |f} | sbt d d ||| | | fDSt ||| | | d S) Nr&zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fcsfdd}|S)Ncs|fSrkr&)r)modulerrr&r'custom_forwardLszOAlignTextEncoder.forward..create_custom_forward..custom_forwardr&)r)r*r)r)r'create_custom_forwardKsz7AlignTextEncoder.forward..create_custom_forwardrrrrHcss|]}|dk r|VqdSrkr&rr&r&r'r9psz+AlignTextEncoder.forward..)rr(rr*cross_attentions) rFrr'ZtrainingloggerZ warning_once enumerater&r#utils checkpointr:r )r8rrrrrr(rrrrrZall_self_attentionsZall_cross_attentionsZnext_decoder_cacherZ layer_moduleZlayer_head_maskr+Z layer_outputsr&rr'rm)sv          zAlignTextEncoder.forward) NNNNNNFFT)rr r!r\r#rnrr$rrrr rmror&r&rhr'r""s. r"cs0eZdZfddZejejdddZZS)AlignTextPoolercs*tt|j|j|_t|_dSrk)r[r\rrrrZTanhrfrgrhr&r'r\s zAlignTextPooler.__init__rcCs(|dddf}||}||}|S)Nr)rrf)r8rZfirst_token_tensor pooled_outputr&r&r'rms  zAlignTextPooler.forwardr r&r&rhr'r1s r1c@s.eZdZdZeZdZdZddZd ddZ d S) AlignPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. alignTcCst|tjtjfr@|jjjd|jjd|j dk r|j j npt|t rrtj |jj|jj j d|j_n>t|tjr|jjjd|jjd|jdk r|jj|j t|tjr|j j |jjddS)zInitialize the weightsg)meanZstdNTg?)rNrrr_weightdataZnormal_rFZinitializer_rangerXZzero_ AlignModelinitZxavier_uniform_text_projectionZ_is_hf_initializedrrrZfill_)r8r)r&r&r' _init_weightss       z"AlignPreTrainedModel._init_weightsFcCst|ttfr||_dSrk)rNAlignTextModelAlignVisionModelr')r8r)rr&r&r'_set_gradient_checkpointingsz0AlignPreTrainedModel._set_gradient_checkpointingN)F) rr r!r"r config_classZbase_model_prefixZsupports_gradient_checkpointingr;r>r&r&r&r'r3s r3z@The text model from ALIGN without any head or projection on top.cseZdZeZdeedfdd ZddZddZe e e e ed de eje eje eje eje eje eje ee ee eeee fd d d ZZS)r<T)rFadd_pooling_layercsDt|||_t||_t||_|r2t|nd|_| dSrk) r[r\rFrrr"encoderr1pooler post_init)r8rFr@rhr&r'r\s    zAlignTextModel.__init__cCs|jjSrkrrr7r&r&r'get_input_embeddingssz#AlignTextModel.get_input_embeddingscCs ||j_dSrkrD)r8rr&r&r'set_input_embeddingssz#AlignTextModel.set_input_embeddings output_typer?N rrrrrrrrrr2c Cs|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd| \} } |dk r|jn|j} |dkrtj | | f| d}|dkr t |j dr|j j ddd| f}| | | }|}ntj| tj| d}||| }|||jj}|j ||||d}|j|||||| d }|d }|jdk r||nd}| s||f|d dSt|||j|j|jd S) a? Returns: Examples: ```python >>> from transformers import AutoTokenizer, AlignTextModel >>> model = AlignTextModel.from_pretrained("kakaobrain/align-base") >>> tokenizer = AutoTokenizer.from_pretrained("kakaobrain/align-base") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsr=rr)rrrr)rrrrrrr)r pooler_outputrr*r,)rFrruse_return_dictrZ%warn_if_padding_and_no_attention_maskrr>r#ZonesrrrrrqrZget_extended_attention_maskZ get_head_maskr%rArBr rr*r,)r8rrrrrrrrrrZ batch_sizerr>rrZextended_attention_maskembedding_outputencoder_outputsZsequence_outputr2r&r&r'rmsb      zAlignTextModel.forward)T) NNNNNNNNN)rr r!rr?rr\rErFrALIGN_TEXT_INPUTS_DOCSTRINGrr rr#rnrrrmror&r&rhr'r<s6   r<zBThe vision model from ALIGN without any head or projection on top.c szeZdZeZdZedfdd ZejdddZ e e e e edd eejeeeeeee fd d d ZZS)r=rjrQcs~t|||_t||_t||_|jdkrDtj |j dd|_ n.|jdkrbtj |j dd|_ nt d|j|dS)Nr5T)Z ceil_moderIz2config.pooling must be one of ['mean', 'max'] got )r[r\rFrPrrrAZ pooling_typerZ AvgPool2dZ hidden_dimrBZ MaxPool2drZpoolingrCrgrhr&r'r\;s     zAlignVisionModel.__init__r1cCs |jjjSrk) vision_modelrr`r7r&r&r'rELsz%AlignVisionModel.get_input_embeddingsrGNrjrrr2cCs|dk r |n|jj}|dk r |n|jj}|dkr8td||}|j|||d}|d}||}||jdd}|s||f|ddSt |||j dS)a Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AlignVisionModel >>> model = AlignVisionModel.from_pretrained("kakaobrain/align-base") >>> processor = AutoProcessor.from_pretrained("kakaobrain/align-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```Nz You have to specify pixel_values)rrrrHr)rrJr) rFrrKrrrArBZreshaperr r)r8rjrrrLrMrr2r&r&r'rmOs*  zAlignVisionModel.forward)NNN)rr r!rr?Zmain_input_namer\rModulerErALIGN_VISION_INPUTS_DOCSTRINGrr rr#r$rrrrmror&r&rhr'r=3s  r=cs,eZdZeZedfdd Zeedee j ee j ee j ee j ee j ee j ee ee ee e j d ddZ eedee j ee ee e j dd d Zeeeeed dee jee j ee j ee j ee j ee j ee j ee ee ee ee eeefd d dZZS)r8rQcst|t|jts.tdt|jdt|jtsPtdt|jd|j}|j}|j |_ |j |_ t ||_ t||_t|j |j |_tt|jj|_|dS)NzLconfig.text_config is expected to be of type AlignTextConfig but is of type .zPconfig.vision_config is expected to be of type AlignVisionConfig but is of type )r[r\rN text_configrrtype vision_configrZprojection_dimrZtext_embed_dimr< text_modelr=rOrrr: Parameterr#rrFZtemperature_init_value temperaturerC)r8rFrTrVrhr&r'r\s$     zAlignModel.__init__NrIc Cs|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |j||||||||| d } | ddddddf} || } | S)a Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`AlignTextModel`]. Examples: ```python >>> from transformers import AutoTokenizer, AlignModel >>> model = AlignModel.from_pretrained("kakaobrain/align-base") >>> tokenizer = AutoTokenizer.from_pretrained("kakaobrain/align-base") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> text_features = model.get_text_features(**inputs) ```N rrrrrrrrrr)rFrrrKrWr:) r8rrrrrrrrr text_outputsrZ text_featuresr&r&r'get_text_featuress$  zAlignModel.get_text_featuresrPcCsD|dk r |n|jj}|dk r |n|jj}|j|||d}|d}|S)a9 Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`AlignVisionModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AlignModel >>> model = AlignModel.from_pretrained("kakaobrain/align-base") >>> processor = AutoProcessor.from_pretrained("kakaobrain/align-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ```Nrjrrr)rFrrKrO)r8rjrrvision_outputsZimage_featuresr&r&r'get_image_featuresszAlignModel.get_image_featuresrG) rrjrrrrr return_lossrrrr2c  Cs.| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |j|| | d} |j||||||| | | d } | d}| ddddddf}||}||jdddd }||jdddd }t || |j }| }d}|rt |}| s||||| | f}|dk r|f|S|St |||||| | d S) a} Returns: Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, AlignModel >>> model = AlignModel.from_pretrained("kakaobrain/align-base") >>> processor = AutoProcessor.from_pretrained("kakaobrain/align-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr]rZrrrHrT)rrZkeepdim)r,r-r.r)rr/r0)rFrrrKrOrWr:Znormr#rrDrYrEr+)r8rrjrrrrrr`rrrr^r[rr)r.r-r,r r&r&r'rmsT)  zAlignModel.forward) NNNNNNNNN)NNN) NNNNNNNNNNN)rr r!rr?r\rrNrr#rnrr$r\rRr_ALIGN_INPUTS_DOCSTRINGrr+rrrrmror&r&rhr'r8sz4,  r8)T)Nr"r dataclassesrtypingrrrrr#Ztorch.utils.checkpointrZ activationsr Zmodeling_outputsr r r r Zmodeling_utilsrZ pytorch_utilsrrrr/rrrrrZconfiguration_alignrrrZ get_loggerrr-Z_CHECKPOINT_FOR_DOCZ_CONFIG_FOR_DOCZ#ALIGN_PRETRAINED_MODEL_ARCHIVE_LISTZALIGN_START_DOCSTRINGrNrRrarr(r+rnrBrErJrKrrOrQrPr_rprwrrrrrrrrr rrrr"r1r3r<r=r8r&r&r&r's|     2 8&('!QKA2Wc!zT