U ,-eG@sdZddlZddlmZmZmZmZmZddlZddl Zddlm Z ddl m Z ddl m Z ddlmZmZmZmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZd d l m!Z!m"Z"m#Z#e$e%Z&dZ'dddddddddddddddddgZ(Gd d!d!e j)Z*zdd"l+m,Z,e,Z*e&-d#Wn4e.k rBYn e/k r`e&0d$YnXe1e*Gd%d&d&e j)Z2Gd'd(d(e j)Z3Gd)d*d*e j)Z4Gd+d,d,e j)Z5Gd-d.d.e j)Z6Gd/d0d0eZ7d1Z8d2Z9ed3e8Gd4d5d5e7Z:Gd6d7d7e j)Z;Gd8d9d9e j)ZGd>d?d?e j)Z?Gd@dAdAe j)Z@dBZAdCZBdDZCedEeAGdFdGdGe7ZDedHeAGdIdJdJe7ZEdS)Kz Pix2Struct modeling fileN)DictListOptionalTupleUnion)nn) checkpoint)ACT2FN)BaseModelOutputBaseModelOutputWithPooling!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)ALL_LAYERNORM_LAYERS) DUMMY_INPUTS DUMMY_MASKadd_start_docstrings%add_start_docstrings_to_model_forwardis_torch_fx_proxyloggingreplace_return_docstrings)Pix2StructConfigPix2StructTextConfigPix2StructVisionConfigrzgoogle/pix2struct-textcaps-basez google/pix2struct-textcaps-largezgoogle/pix2struct-basezgoogle/pix2struct-largezgoogle/pix2struct-ai2d-basezgoogle/pix2struct-ai2d-largez(google/pix2struct-widget-captioning-basez)google/pix2struct-widget-captioning-largez#google/pix2struct-screen2words-basez$google/pix2struct-screen2words-largezgoogle/pix2struct-docvqa-basezgoogle/pix2struct-docvqa-largezgoogle/pix2struct-ocrvqa-basezgoogle/pix2struct-ocrvqa-largezgoogle/pix2struct-chartqa-basez(google/pix2struct-inforgraphics-vqa-basez)google/pix2struct-inforgraphics-vqa-largecs&eZdZdfdd ZddZZS)Pix2StructLayerNormư>cs&ttt||_||_dS)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/pix2struct/modeling_pix2struct.pyr Ms zPix2StructLayerNorm.__init__cCs\|tjdjddd}|t||j}|jjtj tj fkrR||jj}|j|S)NT)Zkeepdim) tor"float32powmeanZrsqrtr%r$dtypefloat16Zbfloat16)r& hidden_statesZvariancer+r+r,forwardUs zPix2StructLayerNorm.forward)r__name__ __module__ __qualname__r r6 __classcell__r+r+r)r,rLsr) FusedRMSNormzWDiscovered apex.normalization.FusedRMSNorm - will use it instead of Pix2StructLayerNormzJDiscovered apex but it failed to load, falling back to Pix2StructLayerNormcs<eZdZdZeddfdd ZejejdddZZ S) Pix2StructVisionEmbeddingsa- Construct the embeddings from patch. In `Pix2Struct` the input is different from classic Vision-transformer models. Here the input is a sequence of `seq_len` flattened patches that also combines padding patches (tokens). Each patch is represented by a vector of `hidden_size` values. NconfigreturncsRtt|j|j|_t|j|j|_ t|j|j|_ t |j |_ dSN)rr rLinearZpatch_embed_hidden_sizer'patch_projection EmbeddingZseq_len row_embeddercolumn_embedderDropout dropout_ratedropoutr&r?r)r+r,r |s  z#Pix2StructVisionEmbeddings.__init__)flattened_patchesr@cCs|dddddf}|dddddf}|ddddddf}||}||}||}|||}||}|S)Nrrr-)longrCrErFrI)r&rKZ row_indicesZ col_indices embeddingsZrow_embeddingsZcol_embeddingsr+r+r,r6s     z"Pix2StructVisionEmbeddings.forward) r8r9r:__doc__rr r"Tensorr6r;r+r+r)r,r=us r=cs&eZdZfddZdddZZS)Pix2StructVisionAttentioncst|j|_|j|_|j|_|j|_|j|j|_ t j |j|j dd|_ t j |j|j dd|_ t j |j|j dd|_t j |j |jdd|_d|_dSNFbias)rr r'd_kvkey_value_proj_dimZnum_attention_headsn_headsZattention_dropoutrI inner_dimrrBquerykeyvalueoutputgradient_checkpointingrJr)r+r,r s z"Pix2StructVisionAttention.__init__NFcs|jdd\}fdd}||}||} ||} t|| dd} |dkrtjdj||f| j | j d}j rj rd|_ |dkrtj|f| j | j d}|dkr||ddddddf|j }n|||j }d|}||dkt| j j} | | 7} t| tt| j j} tjj| d tjd | } tjj| jj d } |dk r| |} t| | }|ddd j}|}|f|f}|r|| f}|S) z& Self-attention block Nr-cs |djjddSZ projectionr.rr- contiguousviewrVrU transposeZstates batch_sizer&r+r,to_projection_shapesz>Pix2StructVisionAttention.forward..to_projection_shaper rdevicer3Tr.)dimr3ptraining) shaperXrYrZr"matmulrazerosrVrgr3r\rk requires_gradr#rhr/Z masked_fillfinfominmaxtensorr functionalsoftmaxr0type_asrIr_r`rWr[)r&r5attention_mask position_biaslayer_head_maskoutput_attentions seq_lengthre query_states key_states value_statesscoresposition_bias_masked attn_weights attn_outputoutputsr+rcr,r6sD    &     z!Pix2StructVisionAttention.forward)NNNFr7r+r+r)r,rPs  rPcs*eZdZedfdd ZddZZS)Pix2StructVisionMlpr?csjttj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t |j |_ t |j |_dSrQrr rrBr'd_ffwi_0wi_1worGrHrIr Z dense_act_fnactrJr)r+r,r s  zPix2StructVisionMlp.__init__cCsz|||}||}||}||}t|jjtjrl|j |jjj krl|jjj tj krl| |jjj }||}|SrA rrrrI isinstancerr$r"rOr3Zint8r/r&r5Z hidden_geluZ hidden_linearr+r+r,r6s   zPix2StructVisionMlp.forward)r8r9r:rr r6r;r+r+r)r,rsrc sdeZdZeddfdd Zd ejeejeejee e ejejfe ejfdddZ Z S) Pix2StructVisionLayerNr>csTt|j|_d|_t||_t||_t|j |j d|_ t|j |j d|_ dS)Nrr() rr Zchunk_size_feed_forwardZ seq_len_dimrP attentionrmlprr'layer_norm_epspre_mlp_layer_normpre_attention_layer_normrJr)r+r,r s   zPix2StructVisionLayer.__init__F)r5rw head_maskrzr@c Csb|}||}|j||||d}|d}|dd}||}||} || |} | f|}|S)N)rwryrzrr)rrrr) r&r5rwrrzZresidualself_attention_outputsattention_outputr layer_outputr+r+r,r6s    zPix2StructVisionLayer.forward)NNF) r8r9r:rr r"rOrboolrrr6r;r+r+r)r,rs rc sVeZdZeddfdd Zd ejeejeejeeee e e fddd Z Z S) Pix2StructVisionEncoderNr>cs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr+)r).0_rr+r, Dsz4Pix2StructVisionEncoder.__init__..F) rr r?r ModuleListrangenum_hidden_layerslayerr\rJr)rr,r As  z Pix2StructVisionEncoder.__init__FT)r5rwrrzoutput_hidden_states return_dictr@cs|rdnd}rdnd}t|jD]\} } |r8||f}|dk rH|| nd} |jr~|jr~fdd} tjj| | ||| } n| ||| } | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nr+csfdd}|S)Ncs|fSrAr+inputs)modulerzr+r,custom_forward\szVPix2StructVisionEncoder.forward..create_custom_forward..custom_forwardr+rrrzrr,create_custom_forward[sz>Pix2StructVisionEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSrAr+rvr+r+r, ssz2Pix2StructVisionEncoder.forward..last_hidden_stater5 attentions) enumeraterr\rkr"utilsrtupler )r&r5rwrrzrrall_hidden_statesZall_self_attentionsi layer_moduleryr layer_outputsr+rr,r6Gs6      zPix2StructVisionEncoder.forward)NNFFT)r8r9r:rr r"rOrrrrr r6r;r+r+r)r,r@s  rc@s0eZdZdZeZeddZddZddZ dS) Pix2StructPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. cCs$tt}tt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r"rsrr)r&rZ input_mask dummy_inputsr+r+r,rs  z&Pix2StructPreTrainedModel.dummy_inputscCs|jj}t|tr(|jj|dnt|tr>t|jtrJ|jj j n|jj }t|jtrh|jj j n|jj }|j jjj d||ddt|j dr|j jdk r|j jj|jjjj d||ddt|jdr|jjdk r|jjj|jjjj d||ddt|jdr|jjdk r|jjjnt|trJt|jtrb|jj j n|jj }t|jtr|jj jn|jj }t|jtr|jj jn|jj}|jjjj d|||dd|jjjj d||dd|jjjj d||dd|jjjj d|||dd|jr|jjjj d||ddn|t|tjrt|jtrp|jj j n|jj }|jjj d||dd|jdk r|jj|jnt|trt|jtr|jj j n|jj }|j jjj d||ddnt|tj!tj"fr\tj#j$|jj%t&j'd|jj(d%|jj)|j_|jdk r|jjnjt|tr|jdk r|jjdnBt|tjr|jjj d|jj(d|jdk r|jj|jdS)zInitialize the weightsg?gg)r2ZstdrSN)*r?Zinitializer_factorrrr$dataZfill_ Pix2StructTextDenseGatedActDenser text_configr'rrZnormal_hasattrrSZzero_rrPix2StructTextAttentionrT num_headsrXrYrZr[has_relative_attention_biasrelative_attention_biasrrDZ padding_idxPix2StructTextModellm_headrBZConv2dinitZ trunc_normal_r/r"r0Zinitializer_ranger3)r&rfactorr'rrUrVr+r+r, _init_weightss                 z'Pix2StructPreTrainedModel._init_weightscCs|jj}|jj}|dkr tdt|rbt|jddd|}tj||dddfgdd}n4| |j}|dddf |dddf<||d<|dkrtd| |d k||S) Nzself.model.config.decoder_start_token_id has to be defined. In Pix2Struct it is usually set to the pad_token_id.See Pix2Struct docs for more information.r.)r.rhr).rz1self.model.config.pad_token_id has to be defined.) r?decoder_start_token_id pad_token_id ValueErrorrr"fullrlcatZ new_zeroscloneZ masked_fill_)r&rrrZshifted_input_idsr+r+r, _shift_rights    z&Pix2StructPreTrainedModel._shift_rightN) r8r9r:rNr config_classpropertyrrrr+r+r+r,r{s  PraL This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`Pix2StructConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: flattened_patches (`torch.FloatTensor` of shape `(batch_size, sequence_length, num_channels x patch_height x patch_width)`): Flattened and padded pixel values. These values can be obtained using [`AutoImageProcessor`]. See [`Pix2StructVisionImageProcessor.__call__`] for details. Check the [original paper](https://arxiv.org/abs/2210.03347) (figure 5) for more details. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding pixel values. Mask values selected in `[0, 1]`: head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zjThe bare Pix2StructVision Model transformer outputting raw hidden-states without any specific head on top.c seZdZeZdZdZdgZedfdd Z de e dd d d Z d d Z eeeefddddZeeeeeddeejeejeejee ee ee eeefdddZZS)Pix2StructVisionModelrKTrrcsDt|||_t||_t||_t|j|j d|_ | dSNr) rr r?r=rMrencoderrr'r layernorm post_initrJr)r+r,r *s    zPix2StructVisionModel.__init__FN)rrZr@cCst|tr||_dSrA)rrr\r&rrZr+r+r,_set_gradient_checkpointing6s z1Pix2StructVisionModel._set_gradient_checkpointingcCs|jjSrA)rMrCr&r+r+r,get_input_embeddings:sz*Pix2StructVisionModel.get_input_embeddings)heads_to_pruner@cCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrZ prune_heads)r&rrZheadsr+r+r, _prune_heads=sz"Pix2StructVisionModel._prune_heads output_typer)rKrwrrzrrr@c Cs|dk r |n|jj}|dk r |n|jj}|dk r4|n|jj}|dkrLtd|dkrh|jdddk}|||jj}| |}|j ||||||d}|d} | | } |s| f} | |ddSt | |j |jdS) a. Returns: Example: ```python >>> import requests >>> from PIL import Image >>> from transformers import AutoProcessor, Pix2StructVisionModel >>> image_processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructVisionModel.from_pretrained("google/pix2struct-textcaps-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 2048, 768] ``` Nz%You have to specify flattened_patchesr.rr)rwrrzrrrr)r?rzruse_return_dictrsumfloat get_head_maskrrMrrr r5r) r&rKrwrrzrrZembedding_outputencoder_outputsZsequence_outputZ head_outputsr+r+r,r6Es8$  zPix2StructVisionModel.forward)F)NNNNNN)r8r9r:rrmain_input_namesupports_gradient_checkpointing_no_split_modulesrr rrrrrintrrr"PIX2STRUCT_VISION_INPUTS_DOCSTRINGrr _CONFIG_FOR_DOCrr"rOrrr6r;r+r+r)r,r s2   rcs*eZdZedfdd ZddZZS)rrcsjttj|j|jdd|_tj|j|jdd|_tj|j|jdd|_t |j |_ t |j |_dSrQrrJr)r+r,r s  z)Pix2StructTextDenseGatedActDense.__init__cCsz|||}||}||}||}t|jjtjrl|j |jjj krl|jjj tj krl| |jjj }||}|SrArrr+r+r,r6s   z(Pix2StructTextDenseGatedActDense.forwardr8r9r:rr r6r;r+r+r)r,rsrcs*eZdZedfdd ZddZZS)Pix2StructTextLayerFFrcs8tt||_t|j|jd|_t |j |_ dSr) rr rDenseReluDenserr'layer_norm_epsilon layer_normrrGrHrIrJr)r+r,r s  zPix2StructTextLayerFF.__init__cCs&||}||}|||}|SrA)rrrI)r&r5Zforwarded_statesr+r+r,r6s  zPix2StructTextLayerFF.forwardrr+r+r)r,rsrcsFeZdZdedfdd Zeddd Zdd d Zdd dZZ S)rFrcst||_|j|_|j|_|j|_|j|_|j|_ |j |_ |j |j|_ t j|j|jdd|_t j|j|jdd|_t j|j|jdd|_t j|j|jdd|_|jrt |j|j |_t|_d|_dSrQ)rr rrelative_attention_num_bucketsrelative_attention_max_distancer'rTrUrrVrHrIrWrrBrXrYrZr[rDrset pruned_headsr\r&r?rr)r+r,r s" z Pix2StructTextAttention.__init__T cCsd}|r4|d}||dktj|7}t|}nt|t| }|d}||k}|t||t||||tj}t|t ||d}|t |||7}|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr-r) r/r"rLabsrqZ zeros_likelogrmathZ full_likewhere)relative_position bidirectional num_buckets max_distanceZrelative_bucketsZ max_exactZis_smallZrelative_position_if_larger+r+r,_relative_position_buckets.  z1Pix2StructTextAttention._relative_position_bucketNc Cs|dkr|jjj}tj|tj|ddddf}tj|tj|ddddf}||}|j|d|j|jd}||}| dddg d}|S)z%Compute binned relative position biasN)r3rgF)rrrr-rr) rr$rgr"ZarangerLrrrZpermuteZ unsqueeze) r& query_length key_lengthrgZcontext_positionZmemory_positionrZrelative_position_bucketvaluesr+r+r, compute_bias s  z$Pix2StructTextAttention.compute_biasc s|jdd\} | } |dk r\t|dkr>tdt|d| |dkrV|djdn|7} |dkrh| n|jd} fddfd d } |}| |j||dk r|dnd}| |j||dk r|dnd}t||d d}|dkrj s6tj dj | | f|j |j d }jrHjrHd |_nj| | |j d}|dk rz|dddd|d dddf}|dk r||}jrt|jd}d|tj<|dd|f}n|}||7}tjj|dd|}tjj|jjd}|dk r||}t||}|dddj} |}|rV||fnd}|f|f|f}| r|||f}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr-z?past_key_value should have 2 past states: keys and values. Got z past statesrrcs |djjddSr]r^rbrcr+r,re<sz.to_projection_shapecsx|dkr||}n|dkr*||}|dk rt|dkrNtj||gdd}n&|jd|jdkrp||}n|}|S)z4projects hidden states correctly to key/query statesNr-rr)r"rrl)r5Z proj_layerkey_value_statespast_key_value)rer+r,project@s z0Pix2StructTextAttention.forward..projectr rfTrgr.rri)!rllenrrXrYrZr"rmrarrnrVrgr3r\rkrorsizerr#listrrrtrurrvrIr_r`rWr[)r&r5maskr rxr ryr use_cacherzr{Zreal_seq_lengthrr r|r}r~rrrrpresent_key_value_staterr+)rdr&rer,r6sx      (     zPix2StructTextAttention.forward)F)Trr)N)NNNNNNFF) r8r9r:rr staticmethodrrr6r;r+r+r)r,rs 0 rcs(eZdZdfdd ZdddZZS) Pix2StructTextLayerSelfAttentionFcs<tt||d|_t|j|jd|_t |j |_ dS)Nrr rr rrrr'rrrrGrHrIrr)r+r,r s z)Pix2StructTextLayerSelfAttention.__init__Nc CsJ||}|j|||||||d} ||| d}|f| dd} | S)N)rrxryr rrzrrrrrI) r&r5rwrxryr rrznormed_hidden_statesrrr+r+r,r6s  z(Pix2StructTextLayerSelfAttention.forward)F)NNNNFFr7r+r+r)r,rs rcs&eZdZfddZdddZZS)!Pix2StructTextLayerCrossAttentioncs<tt|dd|_t|j|jd|_t |j |_ dS)NFrrrrJr)r+r,r s z*Pix2StructTextLayerCrossAttention.__init__NFc  CsN||} |j| |||||||| d } ||| d} | f| dd} | S)N)rr rxryr rrrzrrr)r&r5r rwrxryr rrrzrrrrr+r+r,r6s  z)Pix2StructTextLayerCrossAttention.forward)NNNNFNFr7r+r+r)r,rs rc s(eZdZdfdd Zd ddZZS) Pix2StructTextBlockFcs0tt||d|_t||_t||_dS)Nr)rr rself_attentionrencoder_decoder_attentionrrrr)r+r,r s  zPix2StructTextBlock.__init__NTc  Cs| dk rj|dkrdnd} t| | krPtd| d| dkr:dnddt| d| dd}| dd}nd \}}|j|||||| | d }|dd\}}|dd}|jtjkrt|rt|jj d }tj || |d }|dk }|r|dk r |d j d}nd}|j |||||||| | d }|d }|jtjkrvt|rvt|jj d }tj || |d }|dk r||d}||dd}| |}|jtjkrt|rt|jj d }tj || |d }|f}| r||f|}n||}|S)Nr-zThere should be z past states. z$2 (past / key) for cross attention. zGot z past key / value states)NN)rwrxryr rrzi)rqrrr)r rwrxryr rrrzr)r rrr3r"r4isinfanyrprrclamprlrr)r&r5rwrxencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrycross_attn_layer_head_maskr rrzrZexpected_num_past_key_valuesZself_attn_past_key_valueZcross_attn_past_key_valuerrZattention_outputsZ clamp_valueZdo_cross_attentionrZcross_attention_outputsrr+r+r,r6sn &        zPix2StructTextBlock.forward)F) NNNNNNNNFFTr7r+r+r)r,rsra The Pix2Struct model was proposed in [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://arxiv.org/abs/2210.03347) by Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova. It's an encoder decoder transformer pre-trained in a image-to-text setting. This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config (Union[`Pix2StructConfig`, `Pix2StructTextConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Pix2StructText is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [Pix2StructText Training](./t5#training). attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pix2StructText uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [Pix2StructText Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention layers. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. a Args: flattened_patches (`torch.FloatTensor` of shape `(batch_size, seq_length, hidden_size)`): Flattened pixel patches. the `hidden_size` is obtained by the following formula: `hidden_size` = `num_channels` * `patch_size` * `patch_size` The process of flattening the pixel patches is done by `Pix2StructProcessor`. attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Pix2StructText uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [Pix2StructText Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, `optional`: *hidden_states*, `optional`: *attentions*) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention layers. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss for the decoder. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z)The standalone text decoder of Pix2StructcseZdZeZdgZdgZdZdddZfddZ d d Z d d Z d dZ ddZ ddZeeeeeddeejeejeejeejeejeejeejeeeejeeeeeeeejeeeeejdfefdddZZS)rrzlm_head.weightTFcCst|ttfr||_dSrA)rrrr\rr+r+r,r+sz/Pix2StructTextModel._set_gradient_checkpointingcsttjj|_tfddtj D|_ t jj d|_ tj|_tjjjdd|_|d|_dS)Ncs g|]}tt|dkdqS)rr)rr)rrrr+r,r4sz0Pix2StructTextModel.__init__..rFrR)rr rrD vocab_sizer' embed_tokensrr num_layersrrrfinal_layer_normrGrHrIrBrrr\rJr)rr,r /s zPix2StructTextModel.__init__c Cs|dkrtd|Sd}|D]}d}|D]}||d||jf}q*|dj|djkrtd|djd|djdt|t|krtdt|dt|d||f}q|S) NzHYou might want to consider setting `use_cache=True` to speed up decodingr+rz%reordered_layer_past_states[0] shape z and layer_past_states[0] shape z mismatchedz&length of reordered_layer_past_states z! and length of layer_past_states )loggerwarningZ index_selectr/rgrlrr )r&past_key_valuesZbeam_idxZreordered_decoder_pastZlayer_past_statesZreordered_layer_past_statesZlayer_past_stater+r+r,_reorder_cache@s(  z"Pix2StructTextModel._reorder_cachecCs|jSrAr'rr+r+r,r^sz(Pix2StructTextModel.get_input_embeddingscCs ||_dSrAr.r&Znew_embeddingsr+r+r,set_input_embeddingsasz(Pix2StructTextModel.set_input_embeddingscCs|jSrArrr+r+r,get_output_embeddingsdsz)Pix2StructTextModel.get_output_embeddingscCs ||_dSrAr1r/r+r+r,set_output_embeddingsgsz)Pix2StructTextModel.set_output_embeddingsrN.)rrwr"r# inputs_embedsrcross_attn_head_maskr,rrzrlabelsrr@c, s"dk r n|jjdk r n|jj| dk r4| n|jj} | dk rH| n|jj} |dk rj|dk rjtdnD|dk r|}|d|d}n"|dk r|dd}ntd|dkr|jdk st d||}|\}}|dk r|ddj d|n|}|dkrt j |||j d}|dkrL|dk rL|j d }t j |||j t jd }|dkrfdgt|j}|||}|dk r|\}}}||f}|dkrt j ||j d}||}nd}|||jj}|||jj}rd nd}| rd nd}rd nd}rd nd}d}d}||} tt|j|D]4\}!\}"}#||!}$||!}%| rd|| f}|jr|jrrtd d fdd}&t|&|"| ||||||$|%d }'n|"| ||||||$|%|#d }'d kr|'dd d|'d d}'|'dd\} }(|'d}|dk r.|'r(dnd}r>||(f}r6||'df}|dk r6||'df}q6|| } || } || })| r|| f}d}*| dk r| |)j } t j!ddd}+|+|)"d|)d| "d}*| st#dd|*|)||||fDSt$|*|)||||dS)a Returns: Example: ```python >>> from transformers import AutoProcessor, Pix2StructTextModel >>> processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructTextModel.from_pretrained("google/pix2struct-textcaps-base") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ``` NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer.zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz.create_custom_forward..custom_forwardr+rrzrrr,rsz:Pix2StructTextModel.forward..create_custom_forward) rwrxr"r#r$ryr%r rrzrArr rr2)Z ignore_indexZ reductioncss|]}|dk r|VqdSrAr+rr+r+r,r$sz.Pix2StructTextModel.forward..)losslogitsr,r5rcross_attentions)%r?rrzrrrrr`r'AssertionErrorrlr"r#rgrLr rZget_extended_attention_maskZinvert_attention_maskrr(rIrzipr\rkr*r+rr)rr/rZCrossEntropyLossr_rr ),r&rrwr"r#r4rr5r,rrzrr6rkwargsZ input_shaperdr{Zmask_seq_lengthZencoder_seq_lengthZextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrZencoder_hidden_shapeZencoder_extended_attention_maskZpresent_key_value_statesrZall_attentionsZall_cross_attentionsrxr$r5rrr ryr%rrrr:r9Zloss_fctr+r7r,r6js#  "                    & zPix2StructTextModel.forward)F) NNNNNNNNNNNNN)r8r9r:rrr_tied_weights_keysrrr r-rr0r2r3r PIX2STRUCT_TEXT_INPUTS_DOCSTRINGrr rrr" LongTensor FloatTensorrOrrrr6r;r+r+r)r,r!sT   rzhA conditional generation model with a language modeling head. Can be used for sequence generation tasks.csXeZdZeZdZdgZedfdd ZddZdd Z e j d d d Z d dZ deee jdddZddZddZeeeeeddeejeejeejeejeejeejeejeeeejeeeejeejeejeeeeeeeee eejefdddZ!d eejeejeejdddZ"Z#S)!"Pix2StructForConditionalGenerationrKzdecoder.lm_head.weightrcs8t|t|j|_t|j|_|j|_| dSrA) rr rZ vision_configrrrdecoderZis_vqarrJr)r+r,r Cs    z+Pix2StructForConditionalGeneration.__init__cCs |jSrA)rDrrr+r+r,rNsz7Pix2StructForConditionalGeneration.get_input_embeddingscCs|j|dSrA)rDr0r/r+r+r,r0Qsz7Pix2StructForConditionalGeneration.set_input_embeddings)r@cCs |jSrA)rDr2rr+r+r,r2Tsz8Pix2StructForConditionalGeneration.get_output_embeddingscCs|j|dSrA)rDr3r/r+r+r,r3Wsz8Pix2StructForConditionalGeneration.set_output_embeddingsN)new_num_tokensr@cCs|j|}||jj_|SrA)rDresize_token_embeddingsr?rr&)r&rEZ model_embedsr+r+r,rFZs  z:Pix2StructForConditionalGeneration.resize_token_embeddingscCs|jSrA)rDrr+r+r, get_decoderbsz.Pix2StructForConditionalGeneration.get_decodercCs|jSrA)rrr+r+r, get_encoderesz.Pix2StructForConditionalGeneration.get_encoderr)rKrwrrrdecoder_head_maskr5rr,r6decoder_inputs_embedsrrzrrr@cCsH| dk r | n|jjj} |dk r"|n|jj}|dkrJ|j|||| ||d}nH|rt|tst|dt|dkrt|dndt|dkr|dndd}|d}| dk r|dkr| dkr|| }|dk r|n| |jj  }d|dddf<|j ||| | ||||| | || |d }|s||St |j|j|j|j|j|j|j|j|jd S) a Returns: Example: Inference: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Pix2StructForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("google/pix2struct-textcaps-base") >>> model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-textcaps-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> # autoregressive generation >>> generated_ids = model.generate(**inputs, max_new_tokens=50) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_text) A stop sign is on a street corner. >>> # conditional generation >>> text = "A picture of" >>> inputs = processor(text=text, images=image, return_tensors="pt", add_special_tokens=False) >>> generated_ids = model.generate(**inputs, max_new_tokens=50) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_text) A picture of a stop sign with a red stop sign ``` Training: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Pix2StructForConditionalGeneration >>> processor = AutoProcessor.from_pretrained("google/pix2struct-base") >>> model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-base") >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "A stop sign is on the street corner." >>> inputs = processor(images=image, return_tensors="pt") >>> labels = processor(text=text, return_tensors="pt").input_ids >>> # forward pass >>> outputs = model(**inputs, labels=labels) >>> loss = outputs.loss >>> print(f"{loss.item():.5f}") 5.94282 ```N)rKrwrrzrrrrr-r) rrwr4r,r"r#rr5rrzrr6r) r9r:r,Zdecoder_hidden_statesZdecoder_attentionsr;Zencoder_last_hidden_stater"Zencoder_attentions)r?rrrrrr r rnerrrDrr9r:r,r5rr;r)r&rKrwrrrrIr5rr,r6rJrrzrrr5Zdecoder_outputsr+r+r,r6hsjO z*Pix2StructForConditionalGeneration.forward)rKrwrc KsP|dkrt||j}|dk r6|ddddf}|||| |||||| d S)Nr.) rKrr,rrwrrrIr5r)r"Z ones_liker/rg) r&rrKrwrr,rrIr5rrr>r+r+r,prepare_inputs_for_generationsz@Pix2StructForConditionalGeneration.prepare_inputs_for_generation)N)NNNNNNNNNNNNNNN) NNNNNNNNN)$r8r9r:rrrr?r rr0rModuler2r3rrrDrFrGrHrPIX2STRUCT_INPUTS_DOCSTRINGrrrr"rBrAZ BoolTensorrOrrrr6rLr;r+r+r)r,rC:sz  rC)FrNrtypingrrrrrr"Ztorch.utils.checkpointrrZ activationsr Zmodeling_outputsr r r rrZmodeling_utilsrZ pytorch_utilsrrrrrrrrrZconfiguration_pix2structrrrZ get_loggerr8r*rZ(PIX2STRUCT_PRETRAINED_MODEL_ARCHIVE_LISTrMrZapex.normalizationr<info ImportError Exceptionr+appendr=rPrrrrZ!PIX2STRUCT_VISION_START_DOCSTRINGrrrrrrrrZPIX2STRUCT_START_DOCSTRINGr@rNrrCr+r+r+r,s     $     $_+; r]!$o_Y