U ,-ej@sdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lmZm Z m!Z!m"Z"d dl#m$Z$e"%e&Z'dZ(dZ)dgZ*ddZ+ddZ,ddZ-ddZ.Gdddej/j0Z1Gdddej/j0Z2Gddde j3Z4Gdd d e j3Z5Gd!d"d"e j3Z6Gd#d$d$e j3Z7Gd%d&d&e j3Z8Gd'd(d(e j3Z9Gd)d*d*e j3Z:Gd+d,d,e j3Z;Gd-d.d.e j3ZGd3d4d4eZ?d5Z@d6ZAe d7e@Gd8d9d9e?ZBe d:e@Gd;d<dd>e j3ZDe d?e@Gd@dAdAe?ZEe dBe@GdCdDdDe?ZFe dEe@GdFdGdGe?ZGe dHe@GdIdJdJe?ZHdS)Kz PyTorch YOSO model.N)Path)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardlogging) YosoConfigzuw-madison/yoso-4096rcCs^z>ddlm}dd}|dddg}|d|d d ddlaWd Stk rXdaYd SXdS) Nr)loadcs,ttjjjddfdd|DS)NZkernelsyosocsg|] }|qSr).0fileZ src_folderrg/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/yoso/modeling_yoso.py <sz:load_cuda_kernels..append_root..)r__file__resolveparent)filesrr!r" append_root:sz&load_cuda_kernels..append_rootzfast_lsh_cumulation_torch.cppzfast_lsh_cumulation.cuzfast_lsh_cumulation_cuda.cufast_lsh_cumulationT)verboseF)Ztorch.utils.cpp_extensionrr)lsh_cumulation Exception)rr(Z src_filesrrr"load_cuda_kernels5s r-cCsNt|tr6g}|D]}|s&|}||q|S|sF|}|SdSN) isinstancelistZ is_contiguous contiguousappendZ input_tensorsouttensorrrr" to_contiguousLs  r6cCsLt|tkr6g}|D]}|tjj|dddq|Stjj|dddSdS)N)pdim)typer0r2rZ functional normalizer3rrr"r<Zs  r<c Cst|dkrtdt|dkr0tdtj|d|d|||jd}dtj||jd}t|||d|d||}t|||d|d||}|dk }|dk } tj ||dd } tj | |dd } | | fS) Nr zQuery has incorrect size.zKey has incorrect size.rr7devicerr8r:) lensize ValueErrortorchZrandnr>arangematmulreshapeintsum) querykeynum_hashZhash_lenZrmatZ raise_powZquery_projectionZkey_projectionZ query_binaryZ key_binaryZ query_hashrrr"hashingds$$$  rLc@s$eZdZeddZeddZdS)YosoCumulationc Cs|d}dtt||ddtj|}||dddddf|dddddf}t||} |||||||||_| S)N hash_code_lenrr8)rCacosrE transposemathpisave_for_backwardconfig) ctx query_maskkey_maskrIrJvaluerUrN expectationcumulation_valuerrr"forwardxs(0 zYosoCumulation.forwardcCst|}|j\}}}}}}|j}|d} t||dd|} t| | d|} t| dd| d|} t|dd|} dd| | | dfS)NrNr8rOr7)r6 saved_tensorsrUrCrErQ)rVgradrWrXrZrIrJrYrUrN weighted_exp grad_querygrad_key grad_valuerrr"backwardszYosoCumulation.backwardN__name__ __module__ __qualname__ staticmethodr\rcrrrr"rMws rMc@s$eZdZeddZeddZdS)YosoLSHCumulationc CsX|d|dkrtd|d|dkr8td|d|dkrTtd|d|dkrptd|d|dkrtd|d|dkrtd t|||||g\}}}}}|j}|d }|d } td| } |d rt|||||| |d\} } nt|||| \} } t|| || || |d} |||| | |||||_ | S) Nrz6Query mask and Key mask differ in sizes in dimension 0z3Query mask and Query differ in sizes in dimension 0z1Query mask and Key differ in sizes in dimension 0z8Query mask and Value mask differ in sizes in dimension 0rz,Key and Value differ in sizes in dimension 1r7z,Query and Key differ in sizes in dimension 2rKrN use_fast_hash) rArBr6is_cudarGr+Z fast_hashrLrTrU)rVrWrXrIrJrYrUuse_cudarKrNhashtable_capacityquery_hash_code key_hash_coder[rrr"r\sT   zYosoLSHCumulation.forwardc Csjt|}|j\}}}}}}}|j} |j} | d} td| } | drt|||||| | d} t||||||| d|| | d }t||||||| d|| | d }ndtt || ddt j | }||dddddf|dddddf}t || dd|}t || d|}t | dd| d|}t | dd|} dd||| dfS)NrNr7 lsh_backwardrr8rO) r6r]rUrkrGr+Zlsh_weighted_cumulationrCrPrErQrRrS)rVr^rWrXrnrorIrJrYrUrlrNrmrbr`rarZr_rrr"rcs`     (0zYosoLSHCumulation.backwardNrdrrrr"ris %rics*eZdZdZfddZdddZZS)YosoEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|jd|j|_ t|j |j|_ tj |j|j d|_ t|j|_|jdt|jddddt|dd |_|jd tj|jtj|jjd dddS) N) padding_idxr7Zeps position_ids)rr8F) persistentposition_embedding_typeabsolutetoken_type_idsdtyper>)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_bufferrCrDexpandgetattrrwzerosrurAlongr>selfrU __class__rr"r}s" zYosoEmbeddings.__init__Nc Cs|dk r|}n|dd}|d}|dkrH|jddd|f}|dkrt|dr|jddd|f}||d|}|}ntj|tj|jjd}|dkr| |}| |} || } |j dkr| |} | | 7} | | } || } | S)Nr8rryrrzrx)rAruhasattrryrrCrrr>rrrwrrr) r input_idsryru inputs_embeds input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrrrr"r\ s,        zYosoEmbeddings.forward)NNNNrerfrg__doc__r}r\ __classcell__rrrr"rrs rrcs0eZdZd fdd ZddZd ddZZS) YosoSelfAttentionNcsHt|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|dk r|n|j|_|j|_|j|_|jdk |_|j|_|j|_|j|_|j|j|j|jd|_|jdk rDt j|j|j|jdf|jddfd |jd |_dS) NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())rNrjrKrprr7F)Z in_channelsZ out_channelsZ kernel_sizepaddingbiasgroups)r|r}rnum_attention_headsrrBrGattention_head_size all_head_sizerLinearrIrJrYrZattention_probs_dropout_probrrwuse_expectationrNZ conv_windowuse_convrjrKrp lsh_configZConv2dconvrrUrwrrr"r}-sD    zYosoSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr8rr7rr )rArrviewpermute)rlayerZnew_layer_shaperrr"transpose_for_scoresZs z&YosoSelfAttention.transpose_for_scoresFcCsJ||}|||}|||}||}|jr\|||ddddddf}|\} } } } || | | | }|| | | | }|| | | | }d|d}| d| d| | |  }d} |j sR| | krR| | | | | f}t j |t j||jdgdd}t j |t j||jdgdd}t j |t j||jdgdd}|j sb|jrrt||g\}}|j rt||||||j}nt||||||j}|j s| | kr|ddddd| f}t|}|| | | | }|jr||7}|dd dd }|dd |jf}|j|}|r@||fn|f}|S) N?g@r r=r8r?rr7r rO)rIrrJrYrrrArFsqueezerepeatrGrrCcatrr>trainingr<rMapplyrrirr1rr)r hidden_statesattention_maskoutput_attentionsZmixed_query_layerZ key_layerZ value_layerZ query_layerZconv_value_layer batch_sizeZ num_headsZseq_lenZhead_dimZ gpu_warp_sizeZpad_sizeZ context_layerZnew_context_layer_shapeoutputsrrr"r\_s  "   zYosoSelfAttention.forward)N)NF)rerfrgr}rr\rrrrr"r,s-rcs4eZdZfddZejejejdddZZS)YosoSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNrt) r|r}rrrdenserrrrrrrrr"r}s zYosoSelfOutput.__init__r input_tensorreturncCs&||}||}|||}|Sr.rrrrrrrrr"r\s  zYosoSelfOutput.forwardrerfrgr}rCTensorr\rrrrr"rs rcs0eZdZd fdd ZddZd ddZZS) YosoAttentionNcs.tt||d|_t||_t|_dS)N)rw)r|r}rrroutputset pruned_headsrrrr"r}s  zYosoAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr?)r@rrrrrrrIrJrYrrrunion)rheadsindexrrr" prune_headss zYosoAttention.prune_headsFcCs4||||}||d|}|f|dd}|S)Nrr)rr)rrrrZ self_outputsattention_outputrrrr"r\szYosoAttention.forward)N)NF)rerfrgr}rr\rrrrr"rsrcs0eZdZfddZejejdddZZS)YosoIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSr.) r|r}rrrintermediate_sizerr/ hidden_actstrr intermediate_act_fnrrrr"r}s   zYosoIntermediate.__init__rrcCs||}||}|Sr.)rrrrrrr"r\s  zYosoIntermediate.forwardrrrrr"rs rcs4eZdZfddZejejejdddZZS) YosoOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) r|r}rrrrrrrrrrrrrr"r}s zYosoOutput.__init__rcCs&||}||}|||}|Sr.rrrrr"r\s  zYosoOutput.forwardrrrrr"rs rcs.eZdZfddZd ddZddZZS) YosoLayercsBt|j|_d|_t||_|j|_t||_t ||_ dSNr) r|r}chunk_size_feed_forward seq_len_dimr attentionZadd_cross_attentionr intermediaterrrrrr"r}s   zYosoLayer.__init__NFcCsF|j|||d}|d}|dd}t|j|j|j|}|f|}|S)Nrrr)rrfeed_forward_chunkrr)rrrrZself_attention_outputsrr layer_outputrrr"r\ s  zYosoLayer.forwardcCs||}|||}|Sr.)rr)rrZintermediate_outputrrrr"rs  zYosoLayer.feed_forward_chunk)NF)rerfrgr}r\rrrrrr"rs rcs&eZdZfddZdddZZS) YosoEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr)r)r_rUrr"r# sz(YosoEncoder.__init__..F) r|r}rUrZ ModuleListrangenum_hidden_layersrgradient_checkpointingrrrr"r}s  zYosoEncoder.__init__NFTc s|rdnd}rdnd}t|jD]l\} } |r8||f}|jrh|jrhfdd} tjj| | ||} n | ||} | d}r"|| df}q"|r||f}|stdd|||fDSt|||dS) Nrcsfdd}|S)Ncs|fSr.r)inputs)modulerrr"custom_forward6szJYosoEncoder.forward..create_custom_forward..custom_forwardr)rrr)rr"create_custom_forward5sz2YosoEncoder.forward..create_custom_forwardrrcss|]}|dk r|VqdSr.r)rvrrr" Ksz&YosoEncoder.forward..)last_hidden_stater attentions) enumeraterrrrCutils checkpointtupler ) rrr head_maskroutput_hidden_states return_dictZall_hidden_statesZall_self_attentionsiZ layer_modulerZ layer_outputsrrr"r\#s2       zYosoEncoder.forward)NNFFTrerfrgr}r\rrrrr"rs rcs0eZdZfddZejejdddZZS)YosoPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dSr) r|r}rrrrr/rrr transform_act_fnrrrrrr"r}Us   z$YosoPredictionHeadTransform.__init__rcCs"||}||}||}|Sr.)rrrrrrr"r\^s   z#YosoPredictionHeadTransform.forwardrrrrr"rTs rcs$eZdZfddZddZZS)YosoLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)r) r|r}r transformrrrrdecoder ParameterrCrrrrrr"r}gs   zYosoLMPredictionHead.__init__cCs||}||}|Sr.)rrrrrr"r\ts  zYosoLMPredictionHead.forwardrrrrr"rfs rcs0eZdZfddZejejdddZZS)YosoOnlyMLMHeadcstt||_dSr.)r|r}r predictionsrrrr"r}|s zYosoOnlyMLMHead.__init__)sequence_outputrcCs||}|Sr.)r)rrprediction_scoresrrr"r\s zYosoOnlyMLMHead.forwardrrrrr"r{s rc@s.eZdZdZeZdZdZddZd ddZ d S) YosoPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)ZmeanZstdNr)r/rrweightdataZnormal_rUZinitializer_rangerZzero_r~rsrZfill_)rrrrr" _init_weightss      z!YosoPreTrainedModel._init_weightsFcCst|tr||_dSr.)r/rr)rrrYrrr"_set_gradient_checkpointings z/YosoPreTrainedModel._set_gradient_checkpointingN)F) rerfrgrr config_classZbase_model_prefixZsupports_gradient_checkpointingrrrrrr"rs raG This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`YosoConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z^The bare YOSO Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZddZddZddZee d e e e e d deejeejeejeejeejeejeeeeeeeee fd d dZZS) YosoModelcs2t|||_t||_t||_|dSr.)r|r}rUrrrrencoder post_initrrrr"r}s    zYosoModel.__init__cCs|jjSr.rrrrrr"get_input_embeddingsszYosoModel.get_input_embeddingscCs ||j_dSr.r)rrYrrr"set_input_embeddingsszYosoModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rZheads_to_prunerrrrr" _prune_headsszYosoModel._prune_headsbatch_size, sequence_lengthr output_typerN) rrryrurrrrrrc Cs|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd| \} } |dk r|jn|j} |dkrtj | | f| d}|dkr t |j dr|j j ddd| f}| | | }|}ntj| tj| d}||| }|||jj}|j ||||d}|j|||||| d}|d }| s|f|d dSt||j|j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer8z5You have to specify either input_ids or inputs_embedsr=ryrz)rruryr)rrrrrrr)rrrcross_attentions)rUrruse_return_dictrBZ%warn_if_padding_and_no_attention_maskrAr>rCZonesrrryrrrZget_extended_attention_maskZ get_head_maskrrr rrr)rrrryrurrrrrrrrr>rrZextended_attention_maskZembedding_outputZencoder_outputsrrrr"r\s^      zYosoModel.forward) NNNNNNNNN)rerfrgr}r r r rYOSO_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrCrboolrrr\rrrrr"rs>   rz2YOSO Model with a `language modeling` head on top.cseZdZddgZfddZddZddZee d e e e e d deejeejeejeejeejeejeejeeeeeeeee fd d dZZS)YosoForMaskedLMzcls.predictions.decoder.weightzcls.predictions.decoder.biascs,t|t||_t||_|dSr.)r|r}rrrclsrrrrr"r}Xs   zYosoForMaskedLM.__init__cCs |jjjSr.rrrr rrr"get_output_embeddingsasz%YosoForMaskedLM.get_output_embeddingscCs||jj_dSr.r)rZnew_embeddingsrrr"set_output_embeddingsdsz%YosoForMaskedLM.set_output_embeddingsrrN rrryrurrlabelsrrrrc  Cs| dk r | n|jj} |j|||||||| | d } | d} || } d}|dk rpt}|| d|jj|d}| s| f| dd}|dk r|f|S|St|| | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrryrurrrrrrr8rlosslogitsrr) rUrrrrrrr rr)rrrryrurrrrrrrrrZmasked_lm_lossloss_fctrrrr"r\gs6  zYosoForMaskedLM.forward) NNNNNNNNNN)rerfrgZ_tied_weights_keysr}rrrrrrrr rrrCrrrrr\rrrrr"rTsB   rcs(eZdZdZfddZddZZS)YosoClassificationHeadz-Head for sentence-level classification tasks.csFtt|j|j|_t|j|_t|j|j |_ ||_ dSr.) r|r}rrrrrrr num_labelsout_projrUrrrr"r}s  zYosoClassificationHead.__init__cKsR|dddddf}||}||}t|jj|}||}||}|S)Nr)rrr rUrr&)rfeatureskwargsxrrr"r\s    zYosoClassificationHead.forwardrrrrr"r$s r$zYOSO Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) YosoForSequenceClassificationcs4t||j|_t||_t||_|dSr.)r|r}r%rrr$ classifierrrrrr"r}s    z&YosoForSequenceClassification.__init__rrNrc  Csr| dk r | n|jj} |j|||||||| | d } | d} || } d}|dk r.|jjdkr|jdkrpd|j_n4|jdkr|jtjks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n || |}nN|jjdkrt }|| d|j| d}n|jjdkr.t}|| |}| s^| f| dd}|dk rZ|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrZ regressionZsingle_label_classificationZmulti_label_classificationr8r )rUrrr+Z problem_typer%r{rCrrGr rrrrrrr)rrrryrurrrrrrrrr"r!r#rrrr"r\sT      "     z%YosoForSequenceClassification.forward) NNNNNNNNNN)rerfrgr}rrrrrrrrrCrrrrr\rrrrr"r*s<  r*zYOSO Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) YosoForMultipleChoicecsDt|t||_t|j|j|_t|jd|_| dSr) r|r}rrrrrpre_classifierr+rrrrr"r}s   zYosoForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrNrc  Cs| dk r | n|jj} |dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||| | d } | d} | dddf}||}t|}| |}|d| }d}|dk rLt }|||}| s||f| dd}|dk rx|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr8rOrrr )rUrshaperrArr-rZReLUr+rrrr)rrrryrurrrrrrZ num_choicesrZ hidden_stateZ pooled_outputr"Zreshaped_logitsr!r#rrrr"r\"sP       zYosoForMultipleChoice.forward) NNNNNNNNNN)rerfrgr}rrrrrrrrrCrrrrr\rrrrr"r,s<  r,zYOSO Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) YosoForTokenClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSr.) r|r}r%rrrrrrrrr+rrrrr"r}ss   z#YosoForTokenClassification.__init__rrNrc  Cs | dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk rt}|dk r|ddk}| d|j}t ||dt |j  |}|||}n|| d|j|d}| s| f| dd}|dk r|f|S|St || | j| jdS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr8rr )rUrrrr+rrr%rCwherer5 ignore_indexZtype_asrrr)rrrryrurrrrrrrrr"r!r#Z active_lossZ active_logitsZ active_labelsrrrr"r\~sJ    z"YosoForTokenClassification.forward) NNNNNNNNNN)rerfrgr}rrrrrrrrrCrrrrr\rrrrr"r/ms<  r/zYOSO Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`).cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e e j e ee ee eeee fd ddZZS) YosoForQuestionAnsweringcsBt|d|_|j|_t||_t|j|j|_| dS)Nr7) r|r}r%rrrrr qa_outputsrrrrr"r}s   z!YosoForQuestionAnswering.__init__rrN) rrryrurrstart_positions end_positionsrrrrc  CsD| dk r | n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dk r|dk rt|dkr|d}t|dkr|d}|d}|d|}|d|}t |d}|||}|||}||d}| s.||f| dd}|dk r*|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr8r?)r1r7)r! start_logits end_logitsrr) rUrrr3splitrr@rAclamprrrr)rrrryrurrr4r5rrrrrr"r6r7Z total_lossZ ignored_indexr#Z start_lossZend_lossrrrr"r\sP             z YosoForQuestionAnswering.forward) NNNNNNNNNNN)rerfrgr}rrrrrrrrrCrrrrr\rrrrr"r2s@  r2)IrrRpathlibrtypingrrrrCZtorch.utils.checkpointrZtorch.nnrrr Z activationsr Zmodeling_outputsr r rrrrZmodeling_utilsrZ pytorch_utilsrrrrrrrrZconfiguration_yosorZ get_loggerreloggerrrZ"YOSO_PRETRAINED_MODEL_ARCHIVE_LISTr-r6r<rLZautogradFunctionrMriModulerrrrrrrrrrrrrZYOSO_START_DOCSTRINGrrrr$r*r,r/r2rrrr"s        Z<!8  2oMVVP