U ,-es@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z ddl mZddlmZmZmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZd d l m!Z!e"e#Z$dZ%dgZ&Gdddej'Z(Gdddej'Z)Gdddej'Z*Gdddej'Z+Gdddej'Z,Gdddej'Z-Gdddej'Z.Gdddej'Z/Gd d!d!ej'Z0Gd"d#d#ej'Z1Gd$d%d%eZ2d&Z3d'Z4ed(e3Gd)d*d*e2Z5ed+e3Gd,d-d-e2Z6ed.e3Gd/d0d0e2Z7Gd1d2d2ej'Z8ed3e3Gd4d5d5e2Z9dS)6zPyTorch LiLT model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) LiltConfigrz!SCUT-DLVCLab/lilt-roberta-en-basecs6eZdZfddZd ddZddZdd ZZS) LiltTextEmbeddingscsttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jdddt|dd|_|j|_tj|j|j|jd|_ dS) N padding_idxZeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r EmbeddingZ vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_buffertorcharangeexpandgetattrr!rselfconfig __class__g/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/lilt/modeling_lilt.pyr$2s$ zLiltTextEmbeddings.__init__Nc Cs|dkr2|dk r(|||j|j}n ||}|dk rD|}n|dd}|dkrrtj|tj|j jd}|dkr| |}| |}||}|j dkr| |}||7}||}||}||fS)Nrdtypedevicer")"create_position_ids_from_input_idsrtor>&create_position_ids_from_inputs_embedssizer1zeroslongrr(r+r!r*r,r0) r6 input_idstoken_type_idsr inputs_embeds input_shaper+ embeddingsr*r:r:r;forwardIs*        zLiltTextEmbeddings.forwardcCs2||}tj|dd||}||S)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr1ZcumsumZtype_asrD)r6rErmaskZincremental_indicesr:r:r;r?ms z5LiltTextEmbeddings.create_position_ids_from_input_idscCsN|dd}|d}tj|jd||jdtj|jd}|d|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrrr<r)rBr1r2rrDr>Z unsqueezer3)r6rGrHZsequence_lengthrr:r:r;rAzs z9LiltTextEmbeddings.create_position_ids_from_inputs_embeds)NNNN)__name__ __module__ __qualname__r$rJr?rA __classcell__r:r:r8r;r1s  $ rcs&eZdZfddZdddZZS)LiltLayoutEmbeddingscstt|j|jd|_t|j|jd|_t|j|jd|_t|j|jd|_ |j |_ tj|j |j|j |j d|_tj|j|j|j d|_tj|j|j |jd|_t|j|_dS)Nr)Z in_featuresZ out_featuresr)r#r$rr%Zmax_2d_position_embeddingsr&x_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingsr'rr)channel_shrink_ratiobox_position_embeddingsLinearbox_linear_embeddingsr,r-r.r/r0r5r8r:r;r$s"   zLiltLayoutEmbeddings.__init__Nc CsTzt||dddddf}||dddddf}||dddddf}||dddddf}Wn,tk r}ztd|W5d}~XYnX||dddddf|dddddf}||dddddf|dddddf} tj|||||| gdd} || } ||} | | } | | } | | } | S)Nrrr z;The `bbox` coordinate values should be within 0-1000 range.rrK) rVrW IndexErrorrXrYr1catr]r[r,r0) r6bboxrZleft_position_embeddingsZupper_position_embeddingsZright_position_embeddingsZlower_position_embeddingserXrYZspatial_position_embeddingsr[r:r:r;rJs2 22    zLiltLayoutEmbeddings.forward)NN)rPrQrRr$rJrSr:r:r8r;rTs rTcs2eZdZd fdd Zd ddZd dd ZZS) LiltSelfAttentionNcs^t|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|j|j|j|_t |j|j|j|j|_t |j|j|j|j|_t |j|_|pt|dd|_|jdks0|jd krR|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r!r" relative_keyrelative_key_queryr^r)r#r$r&num_attention_headshasattr ValueErrorrNattention_head_size all_head_sizerr\querykeyvaluerZ layout_query layout_key layout_valuer.Zattention_probs_dropout_probr0r4r!r)r%distance_embedding)r6r7r!r8r:r;r$sB    zLiltSelfAttention.__init__rcCs:|dd|j|j|f}|j|}|ddddS)Nrrr^rr )rBrgrjviewpermute)r6xrZ new_x_shaper:r:r;transpose_for_scoress  z&LiltSelfAttention.transpose_for_scoresFcCs|j|||jd}|j|||jd}|j|||jd}||} |||} |||} || } t | | dd} t || dd}|j dks|j dkrz| d}tj |tj|jddd}tj |tj|jddd}||}|||jd}|j| jd}|j dkrFtd | |}| |} n4|j dkrztd | |}td | |}| ||} | t|j}|t|j|j}||} ||}|dk r||}tjdd |}||}|dk r||}t ||}|d d dd}| dd|j|jf}|j|}|dk rH| |} tjdd | }||}|dk rt||}t || }|d d dd}| dd|jf}|j|}|r||f|fn||ff}|S)N)rvrrerfrr<)r=zbhld,lrd->bhlrzbhrd,lrd->bhlrrKrr^r )rwrqrZrprorlrmrnr1matmulZ transposer!rBr2rDr>rsrrr)r@r=ZeinsummathsqrtrjrZSoftmaxr0rt contiguousrk)r6 hidden_states layout_inputsattention_mask head_maskoutput_attentionsZlayout_value_layerZlayout_key_layerZlayout_query_layerZmixed_query_layerZ key_layerZ value_layerZ query_layerZattention_scoresZlayout_attention_scores seq_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZtmp_attention_scoresZtmp_layout_attention_scoresZlayout_attention_probsZlayout_context_layerZnew_context_layer_shapeZattention_probsZ context_layeroutputsr:r:r;rJsl                  zLiltSelfAttention.forward)N)r)NNF)rPrQrRr$rwrJrSr:r:r8r;rcs $ rccs4eZdZfddZejejejdddZZS)LiltSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) r#r$rr\r&denser,r-r.r/r0r5r8r:r;r$Ms zLiltSelfOutput.__init__r} input_tensorreturncCs&||}||}|||}|SNrr0r,r6r}rr:r:r;rJSs  zLiltSelfOutput.forwardrPrQrRr$r1TensorrJrSr:r:r8r;rLs rcsZeZdZd fdd ZddZd ejejeejeejee e ejddd Z Z S) LiltAttentionNcsRtt||d|_t||_t|_|j}|j|j |_t||_ ||_dS)N)r!) r#r$rcr6routputset pruned_headsr&rZ layout_output)r6r7r!ori_hidden_sizer8r:r;r$[s   zLiltAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)NrrrK)lenrr6rgrjrrrlrmrnrrrkunion)r6headsindexr:r:r; prune_headsgs zLiltAttention.prune_headsFr}r~rrrrc CsT||||||}||dd|}||dd|}||ff|dd} | S)Nrr)r6rr) r6r}r~rrrZ self_outputsattention_outputlayout_attention_outputrr:r:r;rJyszLiltAttention.forward)N)NNF) rPrQrRr$rr1rr FloatTensorboolrrJrSr:r:r8r;rZs rcs0eZdZfddZejejdddZZS)LiltIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r#r$rr\r&intermediate_sizer isinstanceZ hidden_actstrr intermediate_act_fnr5r8r:r;r$s   zLiltIntermediate.__init__r}rcCs||}||}|Sr)rr)r6r}r:r:r;rJs  zLiltIntermediate.forwardrr:r:r8r;rs rcs4eZdZfddZejejejdddZZS) LiltOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r#r$rr\rr&rr,r-r.r/r0r5r8r:r;r$s zLiltOutput.__init__rcCs&||}||}|||}|Srrrr:r:r;rJs  zLiltOutput.forwardrr:r:r8r;rs rcs`eZdZfddZd ejejeejeejeee ejdddZ dd Z d d Z Z S) LiltLayercst|j|_d|_t||_t||_t||_ |j }|j }|j |j |_ |j |j |_ t||_ t||_||_ ||_ dS)Nr)r#r$chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr&rrZlayout_intermediater)r6r7rZori_intermediate_sizer8r:r;r$s      zLiltLayer.__init__NFrc Csr|j|||||d}|dd}|dd}|dd} t|j|j|j|} t|j|j|j|} | | ff| } | S)Nrrr)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r6r}r~rrrZself_attention_outputsrrr layer_outputZlayout_layer_outputr:r:r;rJs0   zLiltLayer.forwardcCs||}|||}|Sr)rrr6rZintermediate_outputrr:r:r;rs  zLiltLayer.feed_forward_chunkcCs||}|||}|Sr)rrrr:r:r;rs  z#LiltLayer.layout_feed_forward_chunk)NNF)rPrQrRr$r1rrrrrrJrrrSr:r:r8r;rs  rc sdeZdZfddZd ejejeejeejeeeeeee e eje fdddZ Z S) LiltEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr:)r).0_r7r:r; sz(LiltEncoder.__init__..F) r#r$r7rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr5r8rr;r$s  zLiltEncoder.__init__NFT)r}r~rrroutput_hidden_states return_dictrcs|rdnd}rdnd} t|jD]\} } |r8||f}|dk rH|| nd} |jr|jrfdd} tjj| | |||| }n| |||| }|dd}|dd}r"| |df} q"|r||f}|stdd||| fDSt||| dS) Nr:csfdd}|S)Ncs|fSrr:)inputs)modulerr:r;custom_forwardszJLiltEncoder.forward..create_custom_forward..custom_forwardr:)rrr)rr;create_custom_forwardsz2LiltEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSrr:)rvr:r:r; &sz&LiltEncoder.forward..)last_hidden_stater} attentions) enumeraterrZtrainingr1utils checkpointtupler )r6r}r~rrrrrZall_hidden_statesZall_self_attentionsiZ layer_moduleZlayer_head_maskrZ layer_outputsr:rr;rJsP         zLiltEncoder.forward)NNFFT)rPrQrRr$r1rrrrrrr rJrSr:r:r8r;rs  rcs0eZdZfddZejejdddZZS) LiltPoolercs*tt|j|j|_t|_dSr)r#r$rr\r&rZTanh activationr5r8r:r;r$8s zLiltPooler.__init__rcCs(|dddf}||}||}|SNr)rr)r6r}Zfirst_token_tensor pooled_outputr:r:r;rJ=s  zLiltPooler.forwardrr:r:r8r;r7s rc@s2eZdZdZeZdZdZgZddZ d ddZ d S) LiltPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. liltTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)ZmeanZstdNg?)rrr\weightdataZnormal_r7Zinitializer_rangeZbiasZzero_r%rr,Zfill_)r6rr:r:r; _init_weightsRs      z!LiltPreTrainedModel._init_weightsFcCst|tr||_dSr)rrr)r6rrnr:r:r;_set_gradient_checkpointingbs z/LiltPreTrainedModel._set_gradient_checkpointingN)F) rPrQrR__doc__r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesrrr:r:r:r;rFsra= This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`LiltConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a- Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) bbox (`torch.LongTensor` of shape `({0}, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare LiLT Model transformer outputting raw hidden-states without any specific head on top.cseZdZdfdd ZddZddZdd Zee d e e e d de eje eje eje eje eje eje eje ee ee eeeeje fd ddZZS) LiltModelTcsNt|||_t||_t||_t||_|rt |j dr,|j jddd| f}|| | }|}ntj | tj |d}||| }|||jj}|j ||||d \}}|j||d }|j|||||| | d }|d }|jdk r||nd}| s||f|d dSt|||j|jdS)a Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)r<)r>rF)rErrFrG)rar)rrrrrrr)rZ pooler_outputr}r)r7rruse_return_dictriZ%warn_if_padding_and_no_attention_maskrBr>r1rCrDZonesrhrIrFr3Zget_extended_attention_maskZ get_head_maskrrrrr r}r)r6rErarrFrrrGrrrrHZ batch_sizerr>Zbuffered_token_type_idsZ buffered_token_type_ids_expandedZextended_attention_maskZembedding_outputZlayout_embedding_outputZencoder_outputssequence_outputrr:r:r;rJsh&        zLiltModel.forward)T) NNNNNNNNNN)rPrQrRr$rrrrLILT_INPUTS_DOCSTRINGformatrr _CONFIG_FOR_DOCrr1rrrrrJrSr:r:r8r;rs:   rz LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZeedeee dd e e j e e j e e je e j e e j e e je e je e j e ee ee eeee j efd ddZZS) LiltForSequenceClassificationcs>t||j|_||_t|dd|_t||_|dSNF)r) r#r$ num_labelsr7rrLiltClassificationHead classifierrr5r8r:r;r$Js   z&LiltForSequenceClassification.__init__rrN rErarrFrrrGlabelsrrrrc  Cs| dk r | n|jj} |j|||||||| | | d } | d} || }d}|dk r<||j}|jjdkr|jdkr~d|j_n4|jdkr|jt j ks|jt j krd|j_nd|j_|jjdkrt }|jdkr|| | }n |||}nN|jjdkrt}||d|j|d}n|jjdkr 1` a classification loss is computed (Cross-Entropy). Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```N rarrFrrrGrrrrrZ regressionZsingle_label_classificationZmulti_label_classificationrr^losslogitsr}r)r7rrrr@r>Z problem_typerr=r1rDrNrsqueezerrsrrr}rr6rErarrFrrrGrrrrrrrrloss_fctrr:r:r;rJUsX,       "     z%LiltForSequenceClassification.forward) NNNNNNNNNNN)rPrQrRr$rrrrrrrr1 LongTensorrrrrrrJrSr:r:r8r;rAs8  rz Lilt Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeedeee dd e e j e e j e e j e e j e e j e e j e e j e e j e ee ee eeee jefd ddZZS) LiltForTokenClassificationcsbt||j|_t|dd|_|jdk r2|jn|j}t||_ t |j |j|_ | dSr)r#r$rrrclassifier_dropoutr/rr.r0r\r&rrr6r7rr8r:r;r$s  z#LiltForTokenClassification.__init__rrNrc  Cs| dk r | n|jj} |j|||||||| | | d } | d} || } || }d}|dk r||j}t}||d|j |d}| s|f| dd}|dk r|f|S|St ||| j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nrrrr^r) r7rrr0rr@r>rrsrrr}rrr:r:r;rJs<)    z"LiltForTokenClassification.forward) NNNNNNNNNNN)rPrQrRr$rrrrrrrr1rrrrrrrJrSr:r:r8r;rs8   rcs(eZdZdZfddZddZZS)rz-Head for sentence-level classification tasks.csTtt|j|j|_|jdk r,|jn|j}t||_ t|j|j |_ dSr) r#r$rr\r&rrr/r.r0rout_projrr8r:r;r$"s   zLiltClassificationHead.__init__cKsL|dddddf}||}||}t|}||}||}|Sr)r0rr1tanhr)r6featureskwargsrur:r:r;rJ+s     zLiltClassificationHead.forward)rPrQrRrr$rJrSr:r:r8r;rs rz Lilt Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee dd e e j e e j e e j e e j e e j e e j e e j e e j e e j e ee ee eeee jefd ddZZS) LiltForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr) r#r$rrrrr\r& qa_outputsrr5r8r:r;r$>s  z!LiltForQuestionAnswering.__init__rrN) rErarrFrrrGstart_positions end_positionsrrrrc  CsR| dk r | n|jj} |j|||||||| | | d } | d}||}|jddd\}}|d}|d}d}|dk r | dk r t|dkr|d}t| dkr| d} |d}| d|}| d|} t |d}|||}||| }||d}| s<||f| dd}|dk r8|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Returns: Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```NrrrrrK)Z ignore_indexr^)r start_logits end_logitsr}r)r7rrrsplitrr|rrBclamprr r}r)r6rErarrFrrrGrrrrrrrrrrZ total_lossZ ignored_indexrZ start_lossZend_lossrr:r:r;rJHsR5           z LiltForQuestionAnswering.forward) NNNNNNNNNNNN)rPrQrRr$rrrrr rrr1rrrrrrrJrSr:r:r8r;r5s<  r):rrztypingrrrr1Ztorch.utils.checkpointrZtorch.nnrrrZ activationsr Zmodeling_outputsr r r rrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_liltrZ get_loggerrPloggerrZ"LILT_PRETRAINED_MODEL_ARCHIVE_LISTModulerrTrcrrrrrrrrZLILT_START_DOCSTRINGrrrrrrr:r:r:r;sh     Y8 5<N!9oa