U 9%e%@sTdZddlZddlmZmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZmZmZmZmZdd lmZdd lmZmZmZmZd d lmZee Z!d Z"dZ#d ddgZ$Gdddej%Z&Gdddej%Z'Gdddej(Z)Gdddej%Z*Gdddej%Z+Gdddej%Z,Gdddej%Z-Gdd d ej%Z.Gd!d"d"ej%Z/Gd#d$d$ej%Z0Gd%d&d&ej%Z1Gd'd(d(ej%Z2Gd)d*d*eZ3d+Z4d,Z5ed-e4Gd.d/d/e3Z6ed0e4Gd1d2d2e3Z7ed3e4Gd4d5d5e3Z8ed6e4Gd7d8d8e3Z9ed9e4Gd:d;d;e3Z:edd>e3Z;dS)?z PyTorch SqueezeBert model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging)SqueezeBertConfigzsqueezebert/squeezebert-uncasedrzsqueezebert/squeezebert-mnliz%squeezebert/squeezebert-mnli-headlesscs*eZdZdZfddZdddZZS)SqueezeBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|j|j|_ t|j |j|_ tj |j |jd|_ t|j|_|jdt|jddddS)N) padding_idxeps position_ids)rF) persistent)super__init__r Embedding vocab_sizeembedding_sizeZ pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNorm hidden_sizelayer_norm_epsDropouthidden_dropout_probdropoutZregister_buffertorchZarangeexpandselfconfig __class__s/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/squeezebert/modeling_squeezebert.pyr!7s zSqueezeBertEmbeddings.__init__Nc Cs|dk r|}n|dd}|d}|dkrH|jddd|f}|dkrftj|tj|jjd}|dkrx||}||}||}|||} | | } | | } | S)Nrrdtypedevice) sizerr.zeroslongr9r%r&r'r(r-) r1 input_idstoken_type_idsr inputs_embeds input_shapeZ seq_lengthr&r' embeddingsr5r5r6forwardGs        zSqueezeBertEmbeddings.forward)NNNN__name__ __module__ __qualname____doc__r!rB __classcell__r5r5r3r6r4s rcs(eZdZdZfddZddZZS) MatMulWrapperz Wrapper for torch.matmul(). This makes flop-counting easier to implement. Note that if you directly call torch.matmul() in your code, the flop counter will typically ignore the flops of the matmul. cstdSN)r r!r1r3r5r6r!fszMatMulWrapper.__init__cCs t||S)a0 :param inputs: two torch tensors :return: matmul of these tensors Here are the typical dimensions found in BERT (the B is optional) mat1.shape: [B, , M, K] mat2.shape: [B, , K, N] output shape: [B, , M, N] )r.matmul)r1Zmat1Zmat2r5r5r6rBiszMatMulWrapper.forwardrCr5r5r3r6rI`s rIc@s"eZdZdZdddZddZdS) SqueezeBertLayerNormz This is a nn.LayerNorm subclass that accepts NCW data layout and performs normalization in the C dimension. N = batch C = channels W = sequence length -q=cCstjj|||ddS)N)Znormalized_shaper)rr(r!)r1r)rr5r5r6r!{szSqueezeBertLayerNorm.__init__cCs*|ddd}tj||}|dddS)Nrr)permuterr(rB)r1xr5r5r6rB~szSqueezeBertLayerNorm.forwardN)rN)rDrErFrGr!rBr5r5r5r6rMts rMcs(eZdZdZfddZddZZS)ConvDropoutLayerNormz8 ConvDropoutLayerNorm: Conv, Dropout, LayerNorm cs8ttj||d|d|_t||_t||_dSNrZ in_channelsZ out_channelsZ kernel_sizegroups) r r!rConv1dconv1drM layernormr+r-)r1cincoutrU dropout_probr3r5r6r!s  zConvDropoutLayerNorm.__init__cCs*||}||}||}||}|SrJ)rWr-rX)r1 hidden_statesZ input_tensorrQr5r5r6rBs    zConvDropoutLayerNorm.forwardrCr5r5r3r6rRs rRcs(eZdZdZfddZddZZS)ConvActivationz* ConvActivation: Conv, Activation cs,ttj||d|d|_t||_dSrS)r r!rrVrWr act)r1rYrZrUr^r3r5r6r!s zConvActivation.__init__cCs||}||SrJ)rWr^)r1rQoutputr5r5r6rBs zConvActivation.forwardrCr5r5r3r6r]s r]cs>eZdZd fdd ZddZddZdd Zd d ZZS) SqueezeBertSelfAttentionrcst||jdkr0td|d|jd|j|_t||j|_|j|j|_tj||d|d|_ tj||d|d|_ tj||d|d|_ t |j |_tjdd|_t|_t|_d S) z config = used for some things; ignored for others (work in progress...) cin = input channels = output channels groups = number of groups to use in conv1d layers rzcin (z6) is not a multiple of the number of attention heads ()rrTrdimN)r r!num_attention_heads ValueErrorintattention_head_size all_head_sizerrVquerykeyvaluer+Zattention_probs_dropout_probr-ZSoftmaxsoftmaxrI matmul_qk matmul_qkv)r1r2rYq_groupsk_groupsv_groupsr3r5r6r!s z!SqueezeBertSelfAttention.__init__cCs:|d|j|j|df}|j|}|ddddS)z - input: [N, C, W] - output: [N, C1, W, C2] where C1 is the head index, and C2 is one head's contents rrrr rO)r:rdrgviewrPr1rQZ new_x_shaper5r5r6transpose_for_scoress  z-SqueezeBertSelfAttention.transpose_for_scorescCs.|d|j|j|df}|j|}|S)z - input: [N, C, W] - output: [N, C1, C2, W] where C1 is the head index, and C2 is one head's contents rr)r:rdrgrrrsr5r5r6transpose_key_for_scoress  z1SqueezeBertSelfAttention.transpose_key_for_scorescCs>|dddd}|d|j|df}|j|}|S)zE - input: [N, C1, W, C2] - output: [N, C, W] rrr rO)rP contiguousr:rhrrrsr5r5r6transpose_outputs z)SqueezeBertSelfAttention.transpose_outputcCs||}||}||}||}||}||} |||} | t|j} | |} | | } | | } | | | } | | } d| i} |r| | d<| S)z expects hidden_states in [N, C, W] data layout. The attention_mask data layout is [N, W], and it does not need to be transposed. context_layerattention_score) rirjrkrtrurmmathsqrtrgrlr-rnrw)r1r\attention_maskoutput_attentionsZmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerryZattention_probsrxresultr5r5r6rBs"           z SqueezeBertSelfAttention.forward)rrr) rDrErFr!rtrurwrBrHr5r5r3r6r`s    r`cs$eZdZfddZddZZS)SqueezeBertModulecst|j}|j}|j}|j}t|||j|j|jd|_t |||j |j d|_ t |||j|jd|_t |||j|j d|_dS)a - hidden_size = input chans = output chans for Q, K, V (they are all the same ... for now) = output chans for the module - intermediate_size = output chans for intermediate layer - groups = number of groups for all layers in the BertModule. (eventually we could change the interface to allow different groups for different layers) )r2rYrorprq)rYrZrUr[)rYrZrUr^N)r r!r)Zintermediate_sizer`rorprq attentionrRZpost_attention_groupsr,post_attentionr]Zintermediate_groups hidden_act intermediateZ output_groupsr_)r1r2Zc0c1c2c3r3r5r6r!s2 zSqueezeBertModule.__init__c CsT||||}|d}|||}||}|||}d|i} |rP|d| d<| S)Nrx feature_mapry)rrrr_) r1r\r|r}ZattZattention_outputZpost_attention_outputZintermediate_output layer_outputZ output_dictr5r5r6rBs    zSqueezeBertModule.forwardrDrErFr!rBrHr5r5r3r6rs rcs&eZdZfddZdddZZS) SqueezeBertEncodercsBtjjkstdtfddtjD|_ dS)NzIf you want embedding_size != intermediate hidden_size, please insert a Conv1d layer to adjust the number of channels before the first SqueezeBertModule.c3s|]}tVqdSrJ)r).0_r2r5r6 5sz.SqueezeBertEncoder.__init__..) r r!r$r)AssertionErrorrZ ModuleListrangenum_hidden_layerslayersr0r3rr6r!,s  zSqueezeBertEncoder.__init__NFTc Cs|dkrd}n|dt|kr&d}nd}|dks:td|ddd}|rPdnd}|r\dnd} |jD]V} |r|ddd}||f7}|ddd}| |||} | d}|rf| | d f7} qf|ddd}|r||f7}|std d ||| fDSt||| d S) NTFzAhead_mask is not yet supported in the SqueezeBert implementation.rrOrr5rrycss|]}|dk r|VqdSrJr5)rvr5r5r6rbsz-SqueezeBertEncoder.forward..)last_hidden_stater\ attentions)countlenrrPrrBtupler ) r1r\r| head_maskr}output_hidden_states return_dictZhead_mask_is_all_noneZall_hidden_statesZall_attentionslayerrr5r5r6rB7s8      zSqueezeBertEncoder.forward)NNFFTrr5r5r3r6r+s rcs$eZdZfddZddZZS)SqueezeBertPoolercs*tt|j|j|_t|_dSrJ)r r!rLinearr)denseZTanh activationr0r3r5r6r!is zSqueezeBertPooler.__init__cCs(|dddf}||}||}|S)Nr)rr)r1r\Zfirst_token_tensor pooled_outputr5r5r6rBns  zSqueezeBertPooler.forwardrr5r5r3r6rhs rcs$eZdZfddZddZZS)"SqueezeBertPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dS)Nr) r r!rrr)r isinstancerstrr transform_act_fnr(r*r0r3r5r6r!xs   z+SqueezeBertPredictionHeadTransform.__init__cCs"||}||}||}|SrJ)rrr(r1r\r5r5r6rBs   z*SqueezeBertPredictionHeadTransform.forwardrr5r5r3r6rws rcs$eZdZfddZddZZS)SqueezeBertLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) r r!r transformrrr)r#decoder Parameterr.r;rr0r3r5r6r!s   z$SqueezeBertLMPredictionHead.__init__cCs||}||}|SrJ)rrrr5r5r6rBs  z#SqueezeBertLMPredictionHead.forwardrr5r5r3r6rs rcs$eZdZfddZddZZS)SqueezeBertOnlyMLMHeadcstt||_dSrJ)r r!r predictionsr0r3r5r6r!s zSqueezeBertOnlyMLMHead.__init__cCs||}|SrJ)r)r1sequence_outputprediction_scoresr5r5r6rBs zSqueezeBertOnlyMLMHead.forwardrr5r5r3r6rs rc@s eZdZdZeZdZddZdS)SqueezeBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformercCst|tjtjfr@|jjjd|jjd|j dk r|j j ndt|tj r|jjjd|jjd|j dk r|jj|j  n$t|t r|j j |jjddS)zInitialize the weightsg)ZmeanZstdNg?)rrrrVweightdataZnormal_r2Zinitializer_rangerZzero_r"rrMZfill_)r1moduler5r5r6 _init_weightss     z(SqueezeBertPreTrainedModel._init_weightsN)rDrErFrGr config_classZbase_model_prefixrr5r5r5r6rsraw The SqueezeBERT model was proposed in [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://arxiv.org/abs/2006.11316) by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. For best results finetuning SqueezeBERT on text classification tasks, it is recommended to use the *squeezebert/squeezebert-mnli-headless* checkpoint as a starting point. Parameters: config ([`SqueezeBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. Hierarchy: ``` Internal class hierarchy: SqueezeBertModel SqueezeBertEncoder SqueezeBertModule SqueezeBertSelfAttention ConvActivation ConvDropoutLayerNorm ``` Data layouts: ``` Input data is in [batch, sequence_length, hidden_size] format. Data inside the encoder is in [batch, hidden_size, sequence_length] format. But, if `output_hidden_states == True`, the data from inside the encoder is returned in [batch, sequence_length, hidden_size] format. The final output of the encoder is in [batch, sequence_length, hidden_size] format. ``` a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zeThe bare SqueezeBERT Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZddZddZddZee d e e e e d deejeejeejeejeejeejeeeeeeeee fd d dZZS)SqueezeBertModelcs6t|t||_t||_t||_|dSrJ) r r!rrArencoderrpooler post_initr0r3r5r6r!$s     zSqueezeBertModel.__init__cCs|jjSrJrAr%rKr5r5r6get_input_embeddings.sz%SqueezeBertModel.get_input_embeddingscCs ||j_dSrJrr1Znew_embeddingsr5r5r6set_input_embeddings1sz%SqueezeBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrZ prune_heads)r1Zheads_to_prunerZheadsr5r5r6 _prune_heads4szSqueezeBertModel._prune_headsbatch_size, sequence_length checkpoint output_typerN) r=r|r>rrr?r}rrreturnc Cs^|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd|dk r|jn|j} |dkrtj | | d}|dkrtj | tj | d}| || } | ||jj}|j||||d} |j| | |||| d}|d}||}| sJ||f|d dSt|||j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embeds)r9r7)r=rr>r?)r\r|rr}rrrr)rZ pooler_outputr\r)r2r}ruse_return_dictreZ%warn_if_padding_and_no_attention_maskr:r9r.Zonesr;r<Zget_extended_attention_maskZ get_head_maskrrArrr r\r)r1r=r|r>rrr?r}rrr@r9Zextended_attention_maskZembedding_outputZencoder_outputsrrr5r5r6rB<sV     zSqueezeBertModel.forward) NNNNNNNNN)rDrErFr!rrrrSQUEEZEBERT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr.TensorZ FloatTensorboolrrrBrHr5r5r3r6rs>   rz9SqueezeBERT Model with a `language modeling` head on top.cseZdZddgZfddZddZddZee d e e e e d deejeejeejeejeejeejeejeeeeeeeee fd d dZZS)SqueezeBertForMaskedLMzcls.predictions.decoder.weightzcls.predictions.decoder.biascs,t|t||_t||_|dSrJ)r r!rrrclsrr0r3r5r6r!s   zSqueezeBertForMaskedLM.__init__cCs |jjjSrJrrrrKr5r5r6get_output_embeddingssz,SqueezeBertForMaskedLM.get_output_embeddingscCs||jj_dSrJrrr5r5r6set_output_embeddingssz,SqueezeBertForMaskedLM.set_output_embeddingsrrN r=r|r>rrr?labelsr}rrrc  Cs| dk r | n|jj} |j|||||||| | d } | d} || } d}|dk rpt}|| d|jj|d}| s| f| dd}|dk r|f|S|St|| | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nr|r>rrr?r}rrrrrOlosslogitsr\r) r2rrrrrrr#r r\r)r1r=r|r>rrr?rr}rroutputsrrZmasked_lm_lossloss_fctr_r5r5r6rBs6  zSqueezeBertForMaskedLM.forward) NNNNNNNNNN)rDrErFZ_tied_weights_keysr!rrrrrrrr rrr.rrrrrBrHr5r5r3r6rsB   rz SqueezeBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) $SqueezeBertForSequenceClassificationcsRt||j|_||_t||_t|j|_ t |j |jj|_ | dSrJ)r r! num_labelsr2rrrr+r,r-rr) classifierrr0r3r5r6r!s  z-SqueezeBertForSequenceClassification.__init__rrNrc  Cs|| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk r8|jjdkr|jdkrzd|j_n4|jdkr|jtj ks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n || |}nN|jjdkrt }|| d|j|d}n|jjdkr8t}|| |}| sh| f| dd}|dk rd|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrZ regressionZsingle_label_classificationZmulti_label_classificationrrOr)r2rrr-rZ problem_typerr8r.r<rfrsqueezerrrrrr\r)r1r=r|r>rrr?rr}rrrrrrrr_r5r5r6rBsV       "     z,SqueezeBertForSequenceClassification.forward) NNNNNNNNNN)rDrErFr!rrrrrrrrr.rrrrrBrHr5r5r3r6rs<  rz SqueezeBERT Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) SqueezeBertForMultipleChoicecs@t|t||_t|j|_t|j d|_ | dS)Nr) r r!rrrr+r,r-rr)rrr0r3r5r6r!?s   z%SqueezeBertForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrNrc  Cst| dk r | n|jj} |dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||| | d } | d} || } || }|d| }d}|dk r0t}|||}| s`|f| dd}|dk r\|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) NrrrrOr) r2rshaperrr:rr-rrrr\r)r1r=r|r>rrr?rr}rrZ num_choicesrrrZreshaped_logitsrrr_r5r5r6rBIsL      z$SqueezeBertForMultipleChoice.forward) NNNNNNNNNN)rDrErFr!rrrrrrrrr.rrrrrBrHr5r5r3r6r7s@  rz SqueezeBERT Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) !SqueezeBertForTokenClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSrJ) r r!rrrrr+r,r-rr)rrr0r3r5r6r!s   z*SqueezeBertForTokenClassification.__init__rrNrc  Cs| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk rxt}|| d|j|d}| s| f| dd}|dk r|f|S|St|| | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. NrrrrOr) r2rrr-rrrrrrr\r)r1r=r|r>rrr?rr}rrrrrrrr_r5r5r6rBs8   z)SqueezeBertForTokenClassification.forward) NNNNNNNNNN)rDrErFr!rrrrrrrrr.rrrrrBrHr5r5r3r6rs<  rz SqueezeBERT Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) SqueezeBertForQuestionAnsweringcs<t||j|_t||_t|j|j|_| dSrJ) r r!rrrrrr) qa_outputsrr0r3r5r6r!s   z(SqueezeBertForQuestionAnswering.__init__rrN) r=r|r>rrr?start_positions end_positionsr}rrrc  CsP| dk r | n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dk r|dk rt|dkr|d}t|dkr|d}|d}| d|}| d|}t |d}|||}|||}||d}| s:||f| dd}|dk r6|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence are not taken into account for computing the loss. Nrrrrrb)Z ignore_indexrO)r start_logits end_logitsr\r)r2rrrsplitrrvrr:clamprrr\r)r1r=r|r>rrr?rrr}rrrrrrrZ total_lossZ ignored_indexrZ start_lossZend_lossr_r5r5r6rBsP           z'SqueezeBertForQuestionAnswering.forward) NNNNNNNNNNN)rDrErFr!rrrrrrrrr.rrrrrBrHr5r5r3r6rs@  r)sv  $   ,Z*= -2cM\VG