U 9%e@s dZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl m Z ddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZmZmZd dlm Z e!e"Z#dZ$dZ%ddddgZ&Gddde j'Z(Gddde j'Z)Gddde j'Z*Gddde j'Z+Gddde j'Z,Gddde j'Z-Gd d!d!e j'Z.Gd"d#d#e j'Z/Gd$d%d%eZ0d&Z1d'Z2ed(e1Gd)d*d*e0Z3Gd+d,d,e j'Z4Gd-d.d.e j'Z5ed/e1Gd0d1d1e0Z6eGd2d3d3eZ7ed4e1Gd5d6d6e0Z8dS)7z PyTorch Splinter model.N) dataclass)ListOptionalTupleUnion)nn)CrossEntropyLoss)ACT2FN))BaseModelOutputWithPastAndCrossAttentions ModelOutputQuestionAnsweringModelOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)SplinterConfigztau/splinter-baserztau/splinter-base-qassztau/splinter-largeztau/splinter-large-qasscsVeZdZdZfddZd eejeejeejeejee e dddZ Z S) SplinterEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jdddt|dd|_dS) N) padding_idxZeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r EmbeddingZ vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_buffertorcharangeexpandgetattrrselfconfig __class__m/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/splinter/modeling_splinter.pyr!4s zSplinterEmbeddings.__init__Nr) input_idstoken_type_idsr inputs_embedspast_key_values_lengthreturnc Cs|dk r|}n|dd}|d}|dkrL|jdd|||f}|dkrjtj|tj|jjd}|dkr|||}||}||} |jdkr| |} | | 7} | | } | | } | S)Nrrdtypedevicer) sizerr.zeroslongr@r%r(rr'r)r-) r3r9r:rr;r< input_shape seq_lengthr( embeddingsr'r7r7r8forwardEs$       zSplinterEmbeddings.forward)NNNNr) __name__ __module__ __qualname____doc__r!rr. LongTensor FloatTensorintrrG __classcell__r7r7r5r8r1s rc seZdZd fdd ZejejdddZd ejeejeejeejeejee e ejee e ejdd d Z Z S) SplinterSelfAttentionNcst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|pt|dd|_|jdks|jd kr|j|_t d |jd |j|_|j|_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()rr relative_keyrelative_key_queryr)r r!r#num_attention_headshasattr ValueErrorrNattention_head_size all_head_sizerLinearquerykeyvaluer+Zattention_probs_dropout_probr-r1rr&r"distance_embedding is_decoderr3r4rr5r7r8r!is* zSplinterSelfAttention.__init__)xr=cCs6|dd|j|jf}||}|ddddS)NrrrTrr )rArUrXviewpermute)r3raZ new_x_shaper7r7r8transpose_for_scoress z*SplinterSelfAttention.transpose_for_scoresF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsr=cCs||}|dk } | r4|dk r4|d} |d} |}n| r^|||} |||} |}nv|dk r|||} |||} tj|d| gdd} tj|d| gdd} n |||} |||} ||} |dk } |jr| | f}t| | dd}|j dks |j dkr| j d| j d}}| r^tj |dtj |j d dd}ntj|tj |j d dd}tj|tj |j d dd}||}|||jd}|j| jd }|j dkrtd | |}||}n4|j dkrtd | |}td | |}|||}|t|j}|dk r:||}tjj|dd}||}|dk rf||}t|| }|dddd }|dd|jf}||}|r||fn|f}|jr||f}|S)NrrrTdimrrRrSr>)r?zbhld,lrd->bhlrzbhrd,lrd->bhlrr ) r[rdr\r]r.catr_matmulZ transposershapeZtensorrCr@rbr/r^r&tor?ZeinsummathsqrtrXrZ functionalZsoftmaxr-rc contiguousrArY)r3rfrgrhrirjrkrlZmixed_query_layerZis_cross_attentionZ key_layerZ value_layerZ query_layer use_cacheZattention_scoresZ query_lengthZ key_lengthZposition_ids_lZposition_ids_rZdistanceZpositional_embeddingZrelative_position_scoresZrelative_position_scores_queryZrelative_position_scores_keyZattention_probsZ context_layerZnew_context_layer_shapeoutputsr7r7r8rGsp             zSplinterSelfAttention.forward)N)NNNNNF) rHrIrJr!r.TensorrdrrMrboolrGrOr7r7r5r8rPhs$rPcs4eZdZfddZejejejdddZZS)SplinterSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) r r!rrZr#denser)r*r+r,r-r2r5r7r8r!s zSplinterSelfOutput.__init__rf input_tensorr=cCs&||}||}|||}|SNr}r-r)r3rfrr7r7r8rGs  zSplinterSelfOutput.forwardrHrIrJr!r.ryrGrOr7r7r5r8r{s r{c sveZdZd fdd ZddZd ejeejeejeejeejee e ejee e ejddd Z Z S) SplinterAttentionNcs.tt||d|_t||_t|_dS)Nr)r r!rPr3r{outputset pruned_headsr`r5r7r8r!s  zSplinterAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrrm)lenrr3rUrXrrr[r\r]rr}rYunion)r3headsindexr7r7r8 prune_headss zSplinterAttention.prune_headsFrec Cs<||||||||}||d|} | f|dd} | S)Nrr)r3r) r3rfrgrhrirjrkrlZ self_outputsattention_outputrxr7r7r8rGs  zSplinterAttention.forward)N)NNNNNF) rHrIrJr!rr.ryrrMrrzrGrOr7r7r5r8rs$rcs0eZdZfddZejejdddZZS)SplinterIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr) r r!rrZr#intermediate_sizer} isinstance hidden_actstrr intermediate_act_fnr2r5r7r8r!1s   zSplinterIntermediate.__init__)rfr=cCs||}||}|Sr)r}r)r3rfr7r7r8rG9s  zSplinterIntermediate.forwardrr7r7r5r8r0s rcs4eZdZfddZejejejdddZZS)SplinterOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr|) r r!rrZrr#r}r)r*r+r,r-r2r5r7r8r!As zSplinterOutput.__init__r~cCs&||}||}|||}|Srrrr7r7r8rGGs  zSplinterOutput.forwardrr7r7r5r8r@s rc steZdZfddZd ejeejeejeejeejeeeejee eejdddZ dd Z Z S) SplinterLayercsrt|j|_d|_t||_|j|_|j|_|jrZ|jsLt|dt|dd|_ t ||_ t ||_ dS)Nrz> should be used as a decoder model if cross attention is addedrr)r r!chunk_size_feed_forward seq_len_dimr attentionr_add_cross_attentionrWcrossattentionr intermediaterrr2r5r7r8r!Ps   zSplinterLayer.__init__NFrec Cs|dk r|ddnd}|j|||||d} | d} |jrP| dd} | d} n | dd} d} |jr|dk rt|dstd|d|dk r|d dnd}|| ||||||}|d} | |dd} |d} | | } t|j|j|j| }|f| } |jr| | f} | S) NrTrlrkrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`ro) rr_rVrWrrfeed_forward_chunkrr)r3rfrgrhrirjrkrlZself_attn_past_key_valueZself_attention_outputsrrxZpresent_key_valueZcross_attn_present_key_valueZcross_attn_past_key_valueZcross_attention_outputs layer_outputr7r7r8rG^sV         zSplinterLayer.forwardcCs||}|||}|Sr)rr)r3rZintermediate_outputrr7r7r8rs  z SplinterLayer.feed_forward_chunk)NNNNNF) rHrIrJr!r.ryrrMrrzrGrrOr7r7r5r8rOs$  ArcseZdZfddZd ejeejeejeejeejeeeejee ee ee ee e eeje fd ddZ Z S) SplinterEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr7)r).0_r4r7r8 sz,SplinterEncoder.__init__..F) r r!r4rZ ModuleListrangenum_hidden_layerslayergradient_checkpointingr2r5rr8r!s  zSplinterEncoder.__init__NFT) rfrgrhrirjpast_key_valuesrwrloutput_hidden_states return_dictr=c  st| rdnd} rdnd} r(|jjr(dnd} |jrJ|jrJ|rJtdd}|rRdnd}t|jD]\}}| rv| |f} |dk r||nd}|dk r||nd|jr|jrևfdd}tj j |||||||}n||||||}|d}|r||df7}r`| |df} |jjr`| |d f} q`| r@| |f} | sbt d d ||| | | fDSt ||| | | d S) Nr7zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fcsfdd}|S)Ncs|fSrr7)inputs)modulerlrkr7r8custom_forwardszNSplinterEncoder.forward..create_custom_forward..custom_forwardr7)rrr)rr8create_custom_forwardsz6SplinterEncoder.forward..create_custom_forwardrrrrTcss|]}|dk r|VqdSrr7)rvr7r7r8 sz*SplinterEncoder.forward..Zlast_hidden_staterrf attentionscross_attentions) r4rrZtrainingloggerZ warning_once enumeraterr.utils checkpointtupler )r3rfrgrhrirjrrwrlrrZall_hidden_statesZall_self_attentionsZall_cross_attentionsZnext_decoder_cacheiZ layer_moduleZlayer_head_maskrZ layer_outputsr7rr8rGsv          zSplinterEncoder.forward) NNNNNNFFT)rHrIrJr!r.ryrrMrrzrr rGrOr7r7r5r8rs. rc@s.eZdZdZeZdZdZddZd ddZ d S) SplinterPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. splinterTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)ZmeanZstdNg?)rrrZweightdataZnormal_r4Zinitializer_rangebiasZzero_r"rr)Zfill_)r3rr7r7r8 _init_weightss      z%SplinterPreTrainedModel._init_weightsFcCst|tr||_dSr)rrr)r3rr]r7r7r8_set_gradient_checkpointing#s z3SplinterPreTrainedModel._set_gradient_checkpointingN)F) rHrIrJrKr config_classZbase_model_prefixZsupports_gradient_checkpointingrrr7r7r7r8rs raK This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`SplinterConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a/ Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `{0}`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `{0}`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `{0}`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Splinter Model transformer outputting raw hidden-states without any specific head on top.cseZdZdZfddZddZddZdd Zee d e e e ed deejeejeejeejeejeejeejeejeeejeeeeeeeeeee fd ddZZS) SplinterModela* The model is an encoder (with only self-attention) following the architecture described in [Attention is all you need](https://arxiv.org/abs/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. cs2t|||_t||_t||_|dSr)r r!r4rrFrencoder post_initr2r5r7r8r!ps    zSplinterModel.__init__cCs|jjSrrFr%)r3r7r7r8get_input_embeddingszsz"SplinterModel.get_input_embeddingscCs ||j_dSrr)r3r]r7r7r8set_input_embeddings}sz"SplinterModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r3Zheads_to_prunerrr7r7r8 _prune_headsszSplinterModel._prune_headsbatch_size, sequence_lengthr output_typerN)r9rgr:rrhr;rirjrrwrlrrr=c Cs| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |jjrZ| dk rP| n|jj} nd} |dk rx|dk rxtdn@|dk r||||}n"|dk r|dd}ntd|\}}|dk r|j n|j }| dk r| ddj dnd}|dkrt j |||f|d}|dkr.t j |t j|d }|||}|jjr|dk r|\}}}||f}|dkr|t j ||d}||}nd}|||jj}|j|||||d }|j|||||| | | | | d }|d}| s|f|d dSt||j|j|j|jd S)a  encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrT)r@r>)r9rr:r;r<) rgrhrirjrrwrlrrrr)r4rlruse_return_dictr_rwrWZ%warn_if_padding_and_no_attention_maskrAr@rrr.ZonesrBrCZget_extended_attention_maskZinvert_attention_maskZ get_head_maskrrFrr rrfrr)r3r9rgr:rrhr;rirjrrwrlrrrD batch_sizerEr@r<Zextended_attention_maskZencoder_batch_sizeZencoder_sequence_lengthrZencoder_hidden_shapeZencoder_extended_attention_maskZembedding_outputZencoder_outputssequence_outputr7r7r8rGsx)         zSplinterModel.forward) NNNNNNNNNNNNN)rHrIrJrKr!rrrrSPLINTER_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr.ryrrMrzrrrGrOr7r7r5r8resP    rcs2eZdZdfdd ZejejdddZZS)SplinterFullyConnectedLayergelucsDt||_||_t|j|j|_t||_t |j|_ dSr) r r! input_dim output_dimrrZr}r act_fnr))r3rrrr5r7r8r! s   z$SplinterFullyConnectedLayer.__init__)rr=cCs"||}||}||}|Sr)r}rr))r3rrfr7r7r8rGs   z#SplinterFullyConnectedLayer.forward)rrr7r7r5r8rs rcs(eZdZdZfddZddZZS)QuestionAwareSpanSelectionHeadzf Implementation of Question-Aware Span Selection (QASS) head, described in Splinter's paper: csztt|j|j|_t|j|j|_t|j|j|_t|j|j|_tj |j|jdd|_ tj |j|jdd|_ dS)NF)r) r r!rr#query_start_transformquery_end_transformstart_transform end_transformrrZstart_classifierend_classifierr2r5r7r8r! s z'QuestionAwareSpanSelectionHead.__init__cCs|\}}}|ddd|}tj|d|d}||}||}||} ||} | |} | ddd} t | | } | |} | ddd} t | | } | | fS)Nrr)rnrrrT) rA unsqueezerepeatr.gatherrrrrrrcrqr)r3r positionsrrnrZ gathered_repsZquery_start_repsZquery_end_repsZ start_repsZend_repsrf start_logits end_logitsr7r7r8rG+s        z&QuestionAwareSpanSelectionHead.forward)rHrIrJrKr!rGrOr7r7r5r8rs rz Splinter Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e je e je ee ee ee e jeee fd ddZZS) SplinterForQuestionAnsweringcs4t|t||_t||_|j|_|dSrr r!rrr splinter_qassquestion_token_idrr2r5r7r8r!Hs    z%SplinterForQuestionAnswering.__init__rrN r9rgr:rrhr;start_positions end_positionsrlrrquestion_positionsr=c  Cs| dk r | n|jj} d} | dkrr|dk rFtjt||jdd}ntj|dtj |j |j d}| d} d} |j ||||||| | | d }|d}||| \}}| r|d |d }}|dk r|d |t|jj}|d |t|jj}d}|dk r|dk rt|d kr0|d}t|d krL|d}|d }|d||d|t|d }|||}|||}||d }| s||f|d d}|dk r|f|S|St||||j|jd S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NFrrmr)r?layoutr@Trgr:rrhr;rlrrrZ ignore_indexrTlossrrrfr)r4rr.ZargmaxeqrrNrBrArCrr@rrrZsqueezefinfor?minrclamp_rr rfr)r3r9rgr:rrhr;rrrlrrrZquestion_positions_were_noneZ"question_position_for_each_examplerxrrr total_lossZ ignored_indexloss_fct start_lossend_lossrr7r7r8rGRsr$           z$SplinterForQuestionAnswering.forward) NNNNNNNNNNNN)rHrIrJr!rrrrrr rrr.ryrLrzrrrGrOr7r7r5r8r@sD  rc@sleZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)SplinterForPreTrainingOutputa Class for outputs of Splinter as a span selection model. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when start and end positions are provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-end scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrrrrfr)rHrIrJrKrrr.rM__annotations__rrrfrrr7r7r7r8rs rz Splinter Model for the recurring span selection task as done during the pretraining. The difference to the QA task is that we do not have a question, but multiple question tokens that replace the occurrences of recurring spans instead. cseZdZfddZeedd eej eej eej eej eej eej eej eej ee ee ee eej e e efd ddZej ej dd d ZZS) SplinterForPreTrainingcs4t|t||_t||_|j|_|dSrrr2r5r7r8r!s    zSplinterForPreTraining.__init__z*batch_size, num_questions, sequence_lengthNrc  Cs| dk r | n|jj} | dkr6|dk r6|dk r6tdn,| dkrP|dkrPtdn| dkrb||} |j||||||| | | d } | d}|\}}}||| \}}| d}|dk r|d|||}|d|t |j j }|d|t |j j }d}|dk r|dk r| dtd|d| dtd|dt|jjd}||||||||}||||||||}||d}| s||f| dd}|dk r|f|S|St|||| j| jd S) a start_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NzCquestion_positions must be specified in order to calculate the lossz>question_positions must be specified when input_embeds is usedrrrrrTr)r4r TypeError_prepare_question_positionsrrArrr0r.rr?rrmaxrr$rbrrfr)r3r9rgr:rrhr;rrrlrrrrxrrZsequence_lengthrnrr num_questionsZ attention_mask_for_each_questionrrrrrr7r7r8rGsl!         zSplinterForPreTraining.forward)r9r=cCslt||jjk\}}t|}tj|d|f|jjtj |j d}t dd|D}||||f<|S)Nrr>cSsg|]}t|qSr7)r.r/)rnr7r7r8r\szFSplinterForPreTraining._prepare_question_positions..) r.wherer4rZbincountfullrArr$rCr@rp)r3r9rowsZflat_positionsrrcolsr7r7r8rSs  z2SplinterForPreTraining._prepare_question_positions) NNNNNNNNNNNN)rHrIrJr!rrrrr.ryrLrzrrrrGrrOr7r7r5r8rs@  dr)9rKrt dataclassesrtypingrrrrr.Ztorch.utils.checkpointrZtorch.nnrZ activationsr Zmodeling_outputsr r r Zmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_splinterrZ get_loggerrHrrrZ&SPLINTER_PRETRAINED_MODEL_ARCHIVE_LISTModulerrPr{rrrrrrZSPLINTER_START_DOCSTRINGrrrrrrrr7r7r7r8sj        72Wb  2 &r