U ,-e @stdZddlZddlmZddlmZmZmZddlZddl Zddlm Z ddl m Z m Z mZddlmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(e&)e*Z+dZ,dZ-dZ.dgZ/ddZ0da1e%rde$rde+2dz e0Wn4e3k r`Z4ze+5de4W5dZ4[4XYnXnddZ6d\ddZ7d]ddZ8d^ddZ9d d!Z:Gd"d#d#ej;j<Z=Gd$d%d%ej;j<Z>Gd&d'd'Z?d_d(d)Z@d*d+ZAd`d,d-ZBGd.d/d/e jCZDGd0d1d1e jCZEGd2d3d3e jCZFGd4d5d5e jCZGGd6d7d7e jCZHGd8d9d9e jCZIGd:d;d;e jCZJGdd?d?e jCZLGd@dAdAe jCZMGdBdCdCe jCZNGdDdEdEeZOdFZPdGZQe"dHePGdIdJdJeOZRe"dKePGdLdMdMeOZSGdNdOdOe jCZTe"dPePGdQdRdReOZUe"dSePGdTdUdUeOZVe"dVePGdWdXdXeOZWe"dYePGdZd[d[eOZXdS)az PyTorch MRA model.N)Path)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)load)ACT2FN)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_ninja_availableis_torch_cuda_availablelogging) MraConfigzuw-madison/mra-base-512-4rZ AutoTokenizercsNttjjjddfdd}|dddg}td|d d ad dladS) NZkernelsmracsfdd|DS)Ncsg|] }|qSr ).0fileZ src_folderr e/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/mra/modeling_mra.py Csz:load_cuda_kernels..append_root..r )filesr#r r$ append_rootBsz&load_cuda_kernels..append_rootzcuda_kernel.cuzcuda_launch.cuztorch_extension.cpp cuda_kernelT)verboser)r__file__resolveparentr r()r'Z src_filesr r#r$load_cuda_kernels>s  r-zLoading custom CUDA kernels...zFailed to load CUDA kernels. Mra requires custom CUDA kernels. Please verify that compatible versions of PyTorch and CUDA Toolkit are installed: cCst|dkrtdt|dkr0td|ddkrFtd|ddkr\td|jd d jd d }|}|}|}t ||||\}}|d d d d d d d d d f}||fS) z8 Computes maximum values for softmax stability. z.sparse_qk_prod must be a 4-dimensional tensor.'indices must be a 2-dimensional tensor. z>The size of the second dimension of sparse_qk_prod must be 32.r z=The size of the third dimension of sparse_qk_prod must be 32.dimN) lensize ValueErrormaxvalues transpose contiguousintr(Z index_max)sparse_qk_prodindicesquery_num_block key_num_blockZ index_valsmax_valsmax_vals_scatterr r r$ sparse_max]s$rDr1cCst|dkrtdt|dkr0td|jd|jdkrLtd|j\}}||}tj|dtj|jd}||||}||dddf||ddf}|S)zN Converts attention mask to a sparse mask for high resolution logits. r/z$mask must be a 2-dimensional tensor.r0rzBmask and indices must have the same size in the zero-th dimension.dtypedeviceN) r6r7r8shapetorcharangelongrGreshape)maskr? block_size batch_sizeseq_len num_block batch_idxr r r$ sparse_maskys &rSc Cs"|\}}}|\}}}||dkr0td||dkrDtd||||||dd}||||||dd}t|dkrtdt|dkrtdt|d krtd |d d krtd |d d krtd|}|}|}|}t|||S)z7 Performs Sampled Dense Matrix Multiplication. rzTquery_size (size of first dimension of dense_query) must be divisible by block_size.Pkey_size (size of first dimension of dense_key) must be divisible by block_size.r5r2r.z+dense_query must be a 4-dimensional tensor.)dense_key must be a 4-dimensional tensor.r/r0r r1z.The third dimension of dense_query must be 32.z,The third dimension of dense_key must be 32.) r7r8rLr;r6r<r=r( mm_to_sparse) dense_query dense_keyr?rNrOZ query_sizer4_key_sizer r r$rVs.  rVc Cs|\}}}||dkr"td|d|kr8td|d|krNtd||||||dd}t|d krtd t|d krtd t|dkrtd |dd krtd|}|}|}|}t||||}|dd||||}|S)zP Performs matrix multiplication of a sparse matrix with a dense matrix. rrTr/zQThe size of the second dimension of sparse_query must be equal to the block_size.r zPThe size of the third dimension of sparse_query must be equal to the block_size.r5r2r.,sparse_query must be a 4-dimensional tensor.rUr0r1z8The size of the third dimension of dense_key must be 32.) r7r8rLr;r6r<r=r(sparse_dense_mm) sparse_queryr?rXr@rNrOrZr4Z dense_qk_prodr r r$r\s. r\cCs |||tj||ddS)NfloorZ rounding_mode)rIdivrK)r?Z dim_1_blockZ dim_2_blockr r r$transpose_indicessrac@s2eZdZeddZeddZed ddZdS) MraSampledDenseMatMulcCs&t||||}||||||_|SN)rVsave_for_backwardrN)ctxrWrXr?rNr>r r r$forwardszMraSampledDenseMatMul.forwardc Csj|j\}}}|j}|d|}|d|}t|||}t|dd|||} t||||} | | ddfSNrr5r2) saved_tensorsrNr7rar\r;) regradrWrXr?rNr@rA indices_Tgrad_key grad_queryr r r$backwards  zMraSampledDenseMatMul.backwardr1cCst||||Src)rbapply)rWrXr?rNr r r$ operator_callsz#MraSampledDenseMatMul.operator_callN)r1__name__ __module__ __qualname__ staticmethodrfrmror r r r$rbs   rbc@s0eZdZeddZeddZeddZdS)MraSparseDenseMatMulcCs&t||||}||||||_|Src)r\rdr@)rer]r?rXr@r>r r r$rfszMraSparseDenseMatMul.forwardc Cs`|j\}}}|j}|d|d}t|||}t|dd|||}t|||} | d|dfSrg)rhr@r7rar\r;rV) rerir]r?rXr@rArjrkrlr r r$rms   zMraSparseDenseMatMul.backwardcCst||||Src)rurn)r]r?rXr@r r r$rosz"MraSparseDenseMatMul.operator_callNrpr r r r$rus   ruc@seZdZeddZdS) MraReduceSumc Cs|\}}}}t|dkr(tdt|dkr@td|\}}}}|\}}|jdd|||}tj|dtj|jd}tj ||dd |dddf|||} tj |||f|j |jd} | d| ||||} | |||} | S) Nr.r[r/r0r3rrEr^r_) r7r6r8sumrLrIrJrKrGr`zerosrFZ index_add) r]r?r@rArOrQrNrYrRZ global_idxestempoutputr r r$ros* & zMraReduceSum.operator_callN)rqrrrsrtror r r r$rvsrvcCs|\}}}||}d} |dk r||||jdd} |||||jdd| dddddfd} |||||jdd| dddddfd} |dk r|||||jdd| dddddfd} nl|tj||tj|jd} |||||jdd} |||||jdd} |dk rD|||||jdd} t| | ddt |} | j dddj }|dk r| d | dddddf| dddddfd k} | | || fS) z/ Compute low resolution approximation. Nr5r3r2ư>rET)r4Zkeepdims@g?)r7rLrwrIonesfloatrGmeanmatmulr;mathsqrtr9r:)querykeyrNrMvaluerOrPhead_dimnum_block_per_row value_hat token_countZ query_hatZkey_hatlow_resolution_logitlow_resolution_logit_row_maxr r r$get_low_resolution_logit0s4  :rcCsT|j\}}}|dkrf|d}tj|||jd} tjtj| | d|d} || dddddfd}|dkr|ddd|ddfd|ddd|ddf<|ddddd|fd|ddddd|f<tj||d|ddd d } | j} |d kr.| j j dd j } || ddddfk }n|d kr>d}nt |d| |fS)zZ Compute the indices of the subset of components to be used in the approximation. rr/rG)ZdiagonalNg@r5TF)r4largestsortedfullr3sparsez# is not a valid approx_model value.) rHrIr}rGZtrilZtriuZtopkrLr?r:minr~r8)r num_blocks approx_modeinitial_prior_first_n_blocksinitial_prior_diagonal_n_blocksrOZtotal_blocks_per_rowrYoffsetZ temp_maskZ diagonal_maskZ top_k_valsr? thresholdhigh_resolution_maskr r r$get_block_idxesXs4    rc $ Cstdkrt|S|\} } } } | | } | |dkrBtd| |}|| | | }|| | | }|| | | }|dk r||dddddf}||dddddf}||dddddf}|dkrt|||||\}}}}n>|dkr"tt||||\}}}}W5QRXnt dt"||}t |||||\}}W5QRXt j ||||dt | }t||||\}}||}|dk r|dd t||dddddddf}t|}t ||||}t ||||}|dkrjt||d||dddddf}t||dddddddfd d |d | | | }|jd d dddddfd d || | }|d d || | |} |dk r| |} t| | dk}!||!dddddf}||!}t| | dk}"||"dddddf}||"}|||dddddf|dddddfd }#n2|dkr||dddddfd }#nt d |dk r|#|dddddf}#|#| | | | }#|#S)z0 Use Mra to approximate self-attention. Nrz4sequence length must be divisible by the block_size.rrz&approx_mode must be "full" or "sparse")rNr|rr5r3r{z-config.approx_mode must be "full" or "sparse")r(rIZ zeros_likeZrequires_grad_r7r8rLrZno_grad ExceptionrrbrorrrDrSexprurvrrepeatrwr~)$rrrrMrrrNrrrOZnum_headrPrZ meta_batchrrrrrrYZlow_resolution_logit_normalizedr?rZhigh_resolution_logitrBrCZhigh_resolution_attnZhigh_resolution_attn_outZhigh_resolution_normalizerZlow_resolution_attnZlow_resolution_attn_outZlow_resolution_normalizerZlog_correctionZlow_resolution_corrZhigh_resolution_corr context_layerr r r$mra2_attention~s     .  $. .   rcs*eZdZdZfddZdddZZS) MraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.csttj|j|j|jd|_t|jd|j|_ t|j |j|_ tj |j|j d|_ t|j|_|dt|jddt|dd|_|jdtj|jtj|jjd d d dS) N) padding_idxr/Zeps position_ids)rr5position_embedding_typeabsolutetoken_type_idsrEF) persistent)super__init__r Embedding vocab_size hidden_sizeZ pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutZregister_bufferrIrJexpandgetattrrrxrr7rKrGselfconfig __class__r r$rs zMraEmbeddings.__init__Nc Cs|dk r|}n|dd}|d}|dkrH|jddd|f}|dkrt|dr|jddd|f}||d|}|}ntj|tj|jjd}|dkr| |}| |} || } |j dkr| |} | | 7} | | } || } | S)Nr5rrrrEr)r7rhasattrrrrIrxrKrGrrrrrr) r input_idsrr inputs_embeds input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrr r r$rfs,        zMraEmbeddings.forward)NNNNrqrrrs__doc__rrf __classcell__r r rr$rs rcs0eZdZdfdd ZddZd ddZZS) MraSelfAttentionNcst|j|jdkr>t|ds>td|jd|jd|j|_t|j|j|_|j|j|_t |j|j|_ t |j|j|_ t |j|j|_ t |j|_|dk r|n|j|_|jd|j|_t|jt|jdd|_|j|_|j|_|j|_dS)NrZembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r1r/)rrrnum_attention_headsrr8r=attention_head_size all_head_sizerLinearrrrrZattention_probs_dropout_probrrrZ block_per_rowrQrrrrrrrrr r$r,s& zMraSelfAttention.__init__cCs6|dd|j|jf}|j|}|ddddS)Nr5rr/rr )r7rrviewpermute)rlayerZnew_layer_shaper r r$transpose_for_scoresHs z%MraSelfAttention.transpose_for_scoresc Cs||}|||}|||}||}|\}}} } d|d}|d|d||| }d} | | kr||| | | f} t j |t j | |j dgdd}t j |t j | |j dgdd}t j |t j | |j dgdd}t |||||j|j|j|jd} | | krB| ddddddd| f} | ||| | } | d d dd } | dd |jf}| j|} | f}|S) N?r|rr1rr5r3)rrrrr/r r2)rrrrr7squeezerrLr=rIcatrxrGrr~rQrrrrr<rr)r hidden_statesattention_maskZmixed_query_layerZ key_layerZ value_layerZ query_layerrOZ num_headsrPrZ gpu_warp_sizeZpad_sizerZnew_context_layer_shapeoutputsr r r$rfMs@       zMraSelfAttention.forward)N)N)rqrrrsrrrfrr r rr$r+srcs4eZdZfddZejejejdddZZS) MraSelfOutputcsBtt|j|j|_tj|j|jd|_t|j |_ dSNr) rrrrrdenserrrrrrrr r$rs zMraSelfOutput.__init__r input_tensorreturncCs&||}||}|||}|Srcrrrrrrr r r$rfs  zMraSelfOutput.forwardrqrrrsrrITensorrfrr r rr$rs rcs0eZdZdfdd ZddZd ddZZS) MraAttentionNcs.tt||d|_t||_t|_dS)N)r)rrrrrrzset pruned_headsrrr r$rs  zMraAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr3)r6rrrrrrrrrrzrrunion)rheadsindexr r r$ prune_headss zMraAttention.prune_headscCs2|||}||d|}|f|dd}|SNrr)rrz)rrrZ self_outputsattention_outputrr r r$rfs zMraAttention.forward)N)N)rqrrrsrrrfrr r rr$rsrcs0eZdZfddZejejdddZZS)MraIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSrc) rrrrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrrr r$rs   zMraIntermediate.__init__rrcCs||}||}|Src)rrrrr r r$rfs  zMraIntermediate.forwardrr r rr$rs rcs4eZdZfddZejejejdddZZS) MraOutputcsBtt|j|j|_tj|j|jd|_t |j |_ dSr) rrrrrrrrrrrrrrr r$rs zMraOutput.__init__rcCs&||}||}|||}|Srcrrr r r$rfs  zMraOutput.forwardrr r rr$rs rcs.eZdZfddZdddZddZZS) MraLayercsBt|j|_d|_t||_|j|_t||_t ||_ dSNr) rrchunk_size_feed_forward seq_len_dimr attentionZadd_cross_attentionr intermediaterrzrrr r$rs   zMraLayer.__init__NcCsB|||}|d}|dd}t|j|j|j|}|f|}|Sr)rrfeed_forward_chunkrr)rrrZself_attention_outputsrr layer_outputr r r$rfs   zMraLayer.forwardcCs||}|||}|Src)rrz)rrZintermediate_outputrr r r$rs  zMraLayer.feed_forward_chunk)N)rqrrrsrrfrrr r rr$rs rcs&eZdZfddZdddZZS) MraEncodercs:t|_tfddtjD|_d|_dS)Ncsg|] }tqSr )r)r!rYrr r$r%sz'MraEncoder.__init__..F) rrrrZ ModuleListrangenum_hidden_layersrgradient_checkpointingrrrr$rs  zMraEncoder.__init__NFTc Cs|rdnd}t|jD]T\}}|r,||f}|jrX|jrXdd} tjj| |||} n |||} | d}q|rz||f}|stdd||fDSt||dS)Nr csfdd}|S)Ncs|Srcr )inputsmoduler r$custom_forwardszIMraEncoder.forward..create_custom_forward..custom_forwardr )rrr rr$create_custom_forwards z1MraEncoder.forward..create_custom_forwardrcss|]}|dk r|VqdSrcr )r!vr r r$ sz%MraEncoder.forward..)last_hidden_stater) enumeraterrZtrainingrIutils checkpointtupler ) rrr head_maskoutput_hidden_states return_dictZall_hidden_statesiZ layer_modulerZ layer_outputsr r r$rfs*      zMraEncoder.forward)NNFTrqrrrsrrfrr r rr$rs  rcs0eZdZfddZejejdddZZS)MraPredictionHeadTransformcsVtt|j|j|_t|jtr6t |j|_ n|j|_ tj |j|j d|_ dSr) rrrrrrrrrr transform_act_fnrrrrr r$rs   z#MraPredictionHeadTransform.__init__rcCs"||}||}||}|Src)rrrrr r r$rf(s   z"MraPredictionHeadTransform.forwardrr r rr$rs rcs$eZdZfddZddZZS)MraLMPredictionHeadcsLtt||_tj|j|jdd|_t t |j|_ |j |j_ dS)NF)bias) rrr transformrrrrdecoder ParameterrIrxrrrr r$r1s   zMraLMPredictionHead.__init__cCs||}||}|Src)rrrr r r$rf>s  zMraLMPredictionHead.forwardrr r rr$r0s rcs0eZdZfddZejejdddZZS)MraOnlyMLMHeadcstt||_dSrc)rrr predictionsrrr r$rFs zMraOnlyMLMHead.__init__)sequence_outputrcCs||}|Src)r)rrprediction_scoresr r r$rfJs zMraOnlyMLMHead.forwardrr r rr$rEs rc@s.eZdZdZeZdZdZddZd ddZ d S) MraPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. rTcCst|tjr:|jjjd|jjd|jdk r|jj nft|tj rz|jjjd|jjd|j dk r|jj|j  n&t|tj r|jj |jj ddS)zInitialize the weightsg)rZstdNr)rrrweightdataZnormal_rZinitializer_rangerZzero_rrrZfill_)rrr r r$ _init_weightsZs      z MraPreTrainedModel._init_weightsFcCst|tr||_dSrc)rrr)rrrr r r$_set_gradient_checkpointingjs z.MraPreTrainedModel._set_gradient_checkpointingN)F) rqrrrsrr config_classZbase_model_prefixZsupports_gradient_checkpointingr!r"r r r r$rPs raF This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MraConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. ak Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. z]The bare MRA Model transformer outputting raw hidden-states without any specific head on top.cseZdZfddZddZddZddZee d e e e e d deejeejeejeejeejeejeeeeeee fd d dZZS)MraModelcs2t|||_t||_t||_|dSrc)rrrrrrencoder post_initrrr r$rs    zMraModel.__init__cCs|jjSrcrrrr r r$get_input_embeddingsszMraModel.get_input_embeddingscCs ||j_dSrcr')rrr r r$set_input_embeddingsszMraModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr%rrr)rZheads_to_prunerrr r r$ _prune_headsszMraModel._prune_headsbatch_size, sequence_lengthr  output_typer#N) rrrrrrrrrc Cs|dk r |n|jj}|dk r |n|jj}|dk rB|dk rBtdn@|dk r`||||} n"|dk rz|dd} ntd| \} } |dk r|jn|j} |dkrtj| | f| d}|dkr t |j dr|j j ddd| f} | | | }|}ntj | tj| d}||| }|||jj}|j ||||d}|j|||||d}|d }|sl|f|d dSt||j|j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer5z5You have to specify either input_ids or inputs_embedsrrrE)rrrr)rrrrrr)r r attentionscross_attentions)rruse_return_dictr8Z%warn_if_padding_and_no_attention_maskr7rGrIr}rrrrrxrKZget_extended_attention_maskZ get_head_maskrr%r rr0r1)rrrrrrrrrrrOrrGrrZextended_attention_maskZembedding_outputZencoder_outputsrr r r$rfsZ       zMraModel.forward)NNNNNNNN)rqrrrsrr)r*r,rMRA_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrIrboolrrrfrr r rr$r$s:   r$z1MRA Model with a `language modeling` head on top.cseZdZddgZfddZddZddZee d e e e e d deejeejeejeejeejeejeejeeeeeee fd d dZZS)MraForMaskedLMzcls.predictions.decoder.weightzcls.predictions.decoder.biascs,t|t||_t||_|dSrc)rrr$rrclsr&rrr r$rs   zMraForMaskedLM.__init__cCs |jjjSrcr9rrr(r r r$get_output_embeddings&sz$MraForMaskedLM.get_output_embeddingscCs||jj_dSrcr:)rZnew_embeddingsr r r$set_output_embeddings)sz$MraForMaskedLM.set_output_embeddingsr-r.N rrrrrrlabelsrrrc  Cs| dk r | n|jj} |j|||||||| d} | d} || } d} |dk rnt}|| d|jj|d} | s| f| dd}| dk r| f|S|St| | | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrrrrrrrrr5rlosslogitsrr0) rr2rr9rrrrrr0)rrrrrrrr>rrrrrZmasked_lm_lossloss_fctrzr r r$rf,s4  zMraForMaskedLM.forward) NNNNNNNNN)rqrrrsZ_tied_weights_keysrr;r<rr3r4rr5rr6rrIrr7rrrfrr r rr$r8s>   r8cs(eZdZdZfddZddZZS)MraClassificationHeadz-Head for sentence-level classification tasks.csFtt|j|j|_t|j|_t|j|j |_ ||_ dSrc) rrrrrrrrr num_labelsout_projrrrr r$ris  zMraClassificationHead.__init__cKsR|dddddf}||}||}t|jj|}||}||}|S)Nr)rrr rrrF)rfeatureskwargsxr r r$rfqs    zMraClassificationHead.forwardrr r rr$rDfs rDzMRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks.cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee eeee fd ddZZS) MraForSequenceClassificationcs4t||j|_t||_t||_|dSrc)rrrEr$rrD classifierr&rrr r$rs    z%MraForSequenceClassification.__init__r-r.Nr=c  Csp| dk r | n|jj} |j|||||||| d} | d} || } d} |dk r,|jjdkr|jdkrnd|j_n4|jdkr|jtjks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | } n || |} nN|jjdkrt }|| d|j| d} n|jjdkr,t}|| |} | s\| f| dd}| dk rX| f|S|St| | | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr?rrZ regressionZsingle_label_classificationZmulti_label_classificationr5r@)rr2rrKZ problem_typerErFrIrKr=r rrrrrrr0)rrrrrrrr>rrrrrBrArCrzr r r$rfsR      "     z$MraForSequenceClassification.forward) NNNNNNNNN)rqrrrsrrr3r4rr5rr6rrIrr7rrrfrr r rr$rJ{s8  rJzMRA Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks.cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee eeee fd ddZZS) MraForMultipleChoicecsDt|t||_t|j|j|_t|jd|_| dSr) rrr$rrrrpre_classifierrKr&rrr r$rs   zMraForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthr.Nr=c  Cs| dk r | n|jj} |dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||| d} | d} | dddf} || } t| } | | }|d| }d}|dk rJt }|||}| sz|f| dd}|dk rv|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr5r2r?rr@)rr2rHrr7rrMrZReLUrKrrrr0)rrrrrrrr>rrZ num_choicesrZ hidden_stateZ pooled_outputrBZreshaped_logitsrArCrzr r r$rfsN       zMraForMultipleChoice.forward) NNNNNNNNN)rqrrrsrrr3r4rr5rr6rrIrr7rrrfrr r rr$rLs8  rLzMRA Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks.cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee eeee fd ddZZS) MraForTokenClassificationcsJt||j|_t||_t|j|_t |j |j|_ | dSrc) rrrEr$rrrrrrrrKr&rrr r$r3s   z"MraForTokenClassification.__init__r-r.Nr=c  Cs| dk r | n|jj} |j|||||||| d} | d} || } || } d} |dk rt}|dk r|ddk}| d|j}t ||dt |j  |}|||} n|| d|j|d} | s| f| dd}| dk r| f|S|St | | | j| jdS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr?rr5rr@)rr2rrrKrrrErIwhereZtensor ignore_indexZtype_asrrr0)rrrrrrrr>rrrrrBrArCZ active_lossZ active_logitsZ active_labelsrzr r r$rf>sH    z!MraForTokenClassification.forward) NNNNNNNNN)rqrrrsrrr3r4rr5rr6rrIrr7rrrfrr r rr$rN-s8  rNzMRA Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`).cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e e j e ee eeee fd ddZZS) MraForQuestionAnsweringcsBt|d|_|j|_t||_t|j|j|_| dS)Nr/) rrrEr$rrrr qa_outputsr&rrr r$rs   z MraForQuestionAnswering.__init__r-r.N) rrrrrrstart_positions end_positionsrrrc  CsB| dk r | n|jj} |j||||||| | d} | d} || } | jddd\}}|d}|d}d}|dk r|dk rt|dkr|d}t|dkr|d}|d}|d|}|d|}t |d}|||}|||}||d}| s,||f| dd}|dk r(|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nr?rrr5r3)rPr/)rA start_logits end_logitsrr0) rr2rrRsplitrr6r7clamprrrr0)rrrrrrrrSrTrrrrrBrUrVZ total_lossZ ignored_indexrCZ start_lossZend_lossrzr r r$rfsN             zMraForQuestionAnswering.forward) NNNNNNNNNN)rqrrrsrrr3r4rr5rr6rrIrr7rrrfrr r rr$rQs<  rQ)r1)r1)r1)NN)r1rr)YrrpathlibrtypingrrrrIZtorch.utils.checkpointrZtorch.nnrrr Ztorch.utils.cpp_extensionr Z activationsr Zmodeling_outputsr rrrrrZmodeling_utilsrZ pytorch_utilsrrrr rrrrrrZconfiguration_mrarZ get_loggerrqloggerr5r6Z_TOKENIZER_FOR_DOCZ!MRA_PRETRAINED_MODEL_ARCHIVE_LISTr-r(inforewarningrDrSrVr\raZautogradFunctionrbrurvrrrModulerrrrrrrrrrrrZMRA_START_DOCSTRINGr3r$r8rDrJrLrNrQr r r r$s             ( ( (- s:V!3  /lLTTN