U 9%e@sddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%d d l&m'Z'e$(e)Z*dZ+dZ,dZ-dZ.dZ/dZ0dZ1dZ2dZ3dZ4dZ5dZ6dZ7dgZ8ddZ9Gddde j:Z;e jGd"d#d#e j:Z?Gd$d%d%e j:Z@Gd&d'd'e j:ZAGd(d)d)e j:ZBGd*d+d+e j:ZCGd,d-d-e j:ZDGd.d/d/e j:ZEGd0d1d1e j:ZFGd2d3d3e j:ZGGd4d5d5e j:ZHGd6d7d7e j:ZIGd8d9d9e j:ZJGd:d;d;e j:ZKGdd?d?e j:ZMGd@dAdAe j:ZNGdBdCdCe j:ZOGdDdEdEeZPeGdFdGdGe ZQdHZRdIZSe"dJeRGdKdLdLePZTe"dMeRGdNdOdOePZUe"dPeRGdQdRdRePZVGdSdTdTe j:ZWe"dUeRGdVdWdWePZXe"dXeRGdYdZdZePZYe"d[eRGd\d]d]ePZZe"d^eRGd_d`d`ePZ[e"daeRGdbdcdcePZ\dS)dN) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)MobileBertConfigzgoogle/mobilebert-uncasedrz mrm8488/mobilebert-finetuned-nerzK['I-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O', 'I-LOC', 'O', 'I-LOC', 'I-LOC']gQ?z#csarron/mobilebert-uncased-squad-v2z'a nice puppet'gףp= @ zlordtt13/emo-mobilebertz'others'z4.72c Cs zddl}ddl}ddl}Wn tk r<tdYnXtj|}t d||j |}g}g} |D]@\} } t d| d| |j || } | | | | qrt|| D]F\} } | dd} | d d } | d d } | d d} | d} tdd| Dr.t dd| q|} | D]}|d|rX|d|}n|g}|ddksz|ddkrt| d} n|ddks|ddkrt| d} nz|ddkrt| d} n`|ddkrt| d} nFzt| |d} Wn2tk r&t dd| Yq6YnXt|dkr6t|d}| |} q6|d dd!krlt| d} n|dkr|| } z,| j| jkstd"| jd#| jd$Wn<tk r}z|j| j| jf7_W5d}~XYnXt d%| t| | _q|S)&z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape Z ffn_layerffnZ FakeLayerNorm LayerNormZextra_output_weightsz dense/kernelZbert mobilebert/css|]}|dkVqdS))Zadam_vZadam_mZAdamWeightDecayOptimizerZAdamWeightDecayOptimizer_1Z global_stepN).0nr%r%q/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/mobilebert/modeling_mobilebert.py ssz0load_tf_weights_in_mobilebert..z Skipping z [A-Za-z]+_\d+z_(\d+)ZkernelgammaweightZ output_biasbetabiasZoutput_weightsZsquad classifierriZ _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpyZ tensorflow ImportErrorloggererrorospathabspathinfotrainZlist_variablesZ load_variableappendzipreplacesplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorchZ from_numpydata)modelconfigZtf_checkpoint_pathr0nptfZtf_pathZ init_varsnamesZarraysnamerFarrayZpointerZm_nameZ scope_namesnumer%r%r(load_tf_weights_in_mobilebertRs                        rTcs2eZdZdfdd ZejejdddZZS)NoNormNcs2ttt||_tt||_dSN) super__init__r ParameterrIzerosr-onesr+)selfZ feat_sizeeps __class__r%r(rXs zNoNorm.__init__) input_tensorreturncCs||j|jSrV)r+r-)r\r`r%r%r(forwardszNoNorm.forward)N__name__ __module__ __qualname__rXrITensorrb __classcell__r%r%r^r(rUsrU) layer_normZno_normcsReZdZdZfddZdeejeejeejeejej dddZ Z S) MobileBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cst|j|_|j|_|j|_tj|j|j|jd|_ t|j |j|_ t|j |j|_ |jrhdnd}|j|}t||j|_t|j|j|_t|j|_|jdt|j ddddS)N) padding_idxr r position_ids)rF) persistent)rWrX trigram_inputembedding_size hidden_sizer Embedding vocab_sizeZ pad_token_idword_embeddingsZmax_position_embeddingsposition_embeddingsZtype_vocab_sizetoken_type_embeddingsLinearembedding_transformationNORM2FNnormalization_typer"Dropouthidden_dropout_probdropoutZregister_bufferrIZarangeexpand)r\rLZembed_dim_multiplierZembedded_input_sizer^r%r(rXs"  zMobileBertEmbeddings.__init__N) input_idstoken_type_idsrl inputs_embedsrac Cs<|dk r|}n|dd}|d}|dkrH|jddd|f}|dkrftj|tj|jjd}|dkrx||}|jrtjt j j |ddddfddddddgdd|t j j |ddddfddddddgddgdd}|js|j |j kr||}||}||}|||} || } || } | S) Nrmrdtypedevicer)valuer/dim)sizerlrIrZlongrrtrocatr functionalpadrprqrxrurvr"r}) r\rrrlr input_shapeZ seq_lengthrurv embeddingsr%r%r(rbs4  ,,      zMobileBertEmbeddings.forward)NNNN) rdrerf__doc__rXrrI LongTensor FloatTensorrgrbrhr%r%r^r(rjs rjc s\eZdZfddZddZd ejejejeejeejee e ejdddZ Z S) MobileBertSelfAttentioncst|j|_t|j|j|_|j|j|_t|j|j|_ t|j|j|_ t|j rf|jn|j |j|_ t|j|_dSrV)rWrXnum_attention_headsrDtrue_hidden_sizeattention_head_size all_head_sizerrwquerykeyuse_bottleneck_attentionrqrr{Zattention_probs_dropout_probr}r\rLr^r%r(rXs z MobileBertSelfAttention.__init__cCs6|dd|j|jf}||}|ddddS)Nrmrr/rr )rrrviewpermute)r\xZ new_x_shaper%r%r(transpose_for_scoress z,MobileBertSelfAttention.transpose_for_scoresN) query_tensor key_tensor value_tensorattention_mask head_maskoutput_attentionsracCs||}||}||} ||} ||} || } t| | dd} | t|j } |dk rp| |} t j j | dd}| |}|dk r||}t|| }|dddd}|dd|jf}||}|r||fn|f}|S)Nrmrrr/rr )rrrrrImatmulrEmathsqrtrrrZsoftmaxr}r contiguousrrr)r\rrrrrrZmixed_query_layerZmixed_key_layerZmixed_value_layerZ query_layerZ key_layerZ value_layerZattention_scoresZattention_probsZ context_layerZnew_context_layer_shapeoutputsr%r%r(rb s(         zMobileBertSelfAttention.forward)NNN) rdrerfrXrrIrgrrboolrrbrhr%r%r^r(rs  rcs4eZdZfddZejejejdddZZS)MobileBertSelfOutputcsTt|j|_t|j|j|_t|j|j|j d|_ |jsPt |j |_ dSNr])rWrXuse_bottleneckrrwrdenseryrzlayer_norm_epsr"r{r|r}rr^r%r(rX4s  zMobileBertSelfOutput.__init__ hidden_statesresidual_tensorracCs,||}|js||}|||}|SrV)rrr}r"r\rr layer_outputsr%r%r(rb<s   zMobileBertSelfOutput.forwardrcr%r%r^r(r3s rc s`eZdZfddZddZd ejejejejeejeejee e ejdddZ Z S) MobileBertAttentioncs*tt||_t||_t|_dSrV)rWrXrr\routputset pruned_headsrr^r%r(rXEs   zMobileBertAttention.__init__cCst|dkrdSt||jj|jj|j\}}t|jj||j_t|jj||j_t|jj ||j_ t|j j |dd|j _ |jjt||j_|jj|jj|j_ |j ||_dS)Nrrr)rCrr\rrrrrrrrrrunion)r\headsindexr%r%r( prune_headsKs zMobileBertAttention.prune_headsN)rrr layer_inputrrrrac Cs:|||||||}||d|} | f|dd} | S)Nrr)r\r) r\rrrrrrrZ self_outputsattention_outputrr%r%r(rb]s  zMobileBertAttention.forward)NNN) rdrerfrXrrIrgrrrrrbrhr%r%r^r(rDs rcs0eZdZfddZejejdddZZS)MobileBertIntermediatecsBtt|j|j|_t|jt r6t |j|_ n|j|_ dSrV) rWrXrrwrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrr^r%r(rXws   zMobileBertIntermediate.__init__rracCs||}||}|SrV)rrr\rr%r%r(rbs  zMobileBertIntermediate.forwardrcr%r%r^r(rvs rcs4eZdZfddZejejejdddZZS)OutputBottleneckcsFtt|j|j|_t|j|j|j d|_ t |j |_ dSr)rWrXrrwrrqrryrzrr"r{r|r}rr^r%r(rXs zOutputBottleneck.__init__rcCs&||}||}|||}|SrV)rr}r"rr%r%r(rbs  zOutputBottleneck.forwardrcr%r%r^r(rs rcs8eZdZfddZejejejejdddZZS)MobileBertOutputcsZt|j|_t|j|j|_t|j |j|_ |jsLt |j |_ n t||_dSrV)rWrXrrrwrrrryrzr"r{r|r}r bottleneckrr^r%r(rXs zMobileBertOutput.__init__)intermediate_statesresidual_tensor_1residual_tensor_2racCsH||}|js*||}|||}n|||}|||}|SrV)rrr}r"r)r\rrr layer_outputr%r%r(rbs   zMobileBertOutput.forwardrcr%r%r^r(rs  rcs0eZdZfddZejejdddZZS)BottleneckLayercs8tt|j|j|_t|j|j|j d|_ dSr) rWrXrrwrqZintra_bottleneck_sizerryrzrr"rr^r%r(rXs zBottleneckLayer.__init__rcCs||}||}|SrVrr")r\rrr%r%r(rbs  zBottleneckLayer.forwardrcr%r%r^r(rs rcs4eZdZfddZejeejdddZZS) Bottleneckcs8t|j|_|j|_t||_|jr4t||_dSrV)rWrXkey_query_shared_bottleneckrrinput attentionrr^r%r(rXs   zBottleneck.__init__rcCsF||}|jr|fdS|jr6||}||||fS||||fSdS)N)rrrr)r\rZbottlenecked_hidden_statesZshared_attention_inputr%r%r(rbs    zBottleneck.forward rdrerfrXrIrgrrbrhr%r%r^r(rs rcs4eZdZfddZejejejdddZZS) FFNOutputcs8tt|j|j|_t|j|j|j d|_ dSr) rWrXrrwrrrryrzrr"rr^r%r(rXs zFFNOutput.__init__rcCs||}|||}|SrVrrr%r%r(rbs zFFNOutput.forwardrcr%r%r^r(rs rcs0eZdZfddZejejdddZZS)FFNLayercs"tt||_t||_dSrV)rWrXr intermediaterrrr^r%r(rXs  zFFNLayer.__init__rcCs||}|||}|SrV)rr)r\rintermediate_outputrr%r%r(rbs  zFFNLayer.forwardrcr%r%r^r(rs rcsLeZdZfddZdejeejeejeee ejdddZ Z S)MobileBertLayercsztj|_j|_t|_t|_t|_ |jrHt |_ jdkrvt fddtjdD|_dS)Nrcsg|] }tqSr%)rr&_rLr%r( sz,MobileBertLayer.__init__..)rWrXrnum_feedforward_networksrrrrrrrrr ModuleListranger!rr^rr(rXs      zMobileBertLayer.__init__N)rrrrrac Cs|jr||\}}}}n|gd\}}}}|j|||||||d} | d} | f} | dd} |jdkrt|jD]\} }|| } | | f7} qr|| }||| |}|f| t d||||| |f| } | S)Nr)rrri) rrrr enumerater!rrrIZtensor)r\rrrrrrrrZself_attention_outputsrsriZ ffn_modulerrr%r%r(rbsJ      zMobileBertLayer.forward)NNN) rdrerfrXrIrgrrrrrbrhr%r%r^r(rs rc sZeZdZfddZd ejeejeejeeeeeee e e fdddZ Z S) MobileBertEncodercs.ttfddtjD|_dS)Ncsg|] }tqSr%)rrrr%r(r5sz.MobileBertEncoder.__init__..)rWrXrrrnum_hidden_layerslayerrr^rr(rX3s zMobileBertEncoder.__init__NFT)rrrroutput_hidden_states return_dictrac Cs|rdnd}|rdnd}t|jD]B\} } |r8||f}| |||| |} | d}|r"|| df}q"|rt||f}|stdd|||fDSt|||dS)Nr%rrcss|]}|dk r|VqdSrVr%)r&vr%r%r(r)Vsz,MobileBertEncoder.forward..)last_hidden_stater attentions)rrtupler ) r\rrrrrrZall_hidden_statesZall_attentionsrZ layer_modulerr%r%r(rb7s.    zMobileBertEncoder.forward)NNFFT)rdrerfrXrIrgrrrrrr rbrhr%r%r^r(r2s  rcs0eZdZfddZejejdddZZS)MobileBertPoolercs.t|j|_|jr*t|j|j|_dSrV)rWrXZclassifier_activation do_activaterrwrqrrr^r%r(rX]s zMobileBertPooler.__init__rcCs6|dddf}|js|S||}t|}|SdS)Nr)rrrItanh)r\rZfirst_token_tensor pooled_outputr%r%r(rbcs   zMobileBertPooler.forwardrcr%r%r^r(r\s rcs0eZdZfddZejejdddZZS)!MobileBertPredictionHeadTransformcsXtt|j|j|_t|jtr6t |j|_ n|j|_ t d|j|j d|_ dS)Nrir)rWrXrrwrqrrrrr transform_act_fnryrr"rr^r%r(rXps   z*MobileBertPredictionHeadTransform.__init__rcCs"||}||}||}|SrV)rrr"rr%r%r(rbys   z)MobileBertPredictionHeadTransform.forwardrcr%r%r^r(ros rcs0eZdZfddZejejdddZZS)MobileBertLMPredictionHeadcshtt||_tj|j|j|jdd|_ tj|j|jdd|_ t t |j|_|j|j _dS)NF)r-)rWrXr transformrrwrsrqrprdecoderrYrIrZr-rr^r%r(rXs   z#MobileBertLMPredictionHead.__init__rcCs>||}|tj|jj|jjgdd}||jj7}|S)Nrr) rrrIrrr+trr-rr%r%r(rbs $ z"MobileBertLMPredictionHead.forwardrcr%r%r^r(rs rcs0eZdZfddZejejdddZZS)MobileBertOnlyMLMHeadcstt||_dSrV)rWrXr predictionsrr^r%r(rXs zMobileBertOnlyMLMHead.__init__)sequence_outputracCs||}|SrV)r)r\rprediction_scoresr%r%r(rbs zMobileBertOnlyMLMHead.forwardrcr%r%r^r(rs rcs8eZdZfddZejejeejdddZZS)MobileBertPreTrainingHeadscs(tt||_t|jd|_dSNr/)rWrXrrrrwrqseq_relationshiprr^r%r(rXs  z#MobileBertPreTrainingHeads.__init__)rrracCs||}||}||fSrV)rr)r\rrrseq_relationship_scorer%r%r(rbs  z"MobileBertPreTrainingHeads.forwardrr%r%r^r(rs rc@s(eZdZdZeZeZeZ dZ ddZ dS)MobileBertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r#cCst|tjr:|jjjd|jjd|jdk r|jj njt|tj rz|jjjd|jjd|j dk r|jj|j  n*t|tj t fr|jj |jjddS)zInitialize the weightsr)ZmeanZstdNg?)rrrwr+rJZnormal_rLZinitializer_ranger-Zzero_rrrkr"rUZfill_)r\moduler%r%r( _init_weightss     z'MobileBertPreTrainedModel._init_weightsN) rdrerfrr config_class(MOBILEBERT_PRETRAINED_MODEL_ARCHIVE_LISTZpretrained_model_archive_maprTZload_tf_weightsZbase_model_prefixrr%r%r%r(rs rc@sleZdZUdZdZeejed<dZ ejed<dZ ejed<dZ ee ejed<dZ ee ejed<dS)MobileBertForPreTrainingOutputab Output type of [`MobileBertForPreTraining`]. Args: loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nlossprediction_logitsseq_relationship_logitsrr)rdrerfrr rrIr__annotations__r r rrrr%r%r%r(rs raD This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MobileBertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a5 Args: input_ids (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `({0})`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zdThe bare MobileBert Model transformer outputting raw hidden-states without any specific head on top.cseZdZdZdfdd ZddZddZd d Zee d e e e ed deejeejeejeejeejeejeeeeeeeee fd ddZZS)MobileBertModelz. https://arxiv.org/pdf/2004.02984.pdf TcsDt|||_t||_t||_|r2t|nd|_| dSrV) rWrXrLrjrrencoderrpooler post_init)r\rLadd_pooling_layerr^r%r(rX2s    zMobileBertModel.__init__cCs|jjSrVrrtr\r%r%r(get_input_embeddings=sz$MobileBertModel.get_input_embeddingscCs ||j_dSrVr)r\rr%r%r(set_input_embeddings@sz$MobileBertModel.set_input_embeddingscCs*|D]\}}|jj|j|qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)r\Zheads_to_prunerrr%r%r( _prune_headsCszMobileBertModel._prune_headsbatch_size, sequence_length checkpoint output_typerN) rrrrlrrrrrrac Csn|dk r |n|jj}|dk r |n|jj}| dk r4| n|jj} |dk rV|dk rVtdn@|dk rt||||} n"|dk r|dd} ntd|dk r|jn|j} |dkrtj | | d}|dkrtj | tj | d}| || } | ||jj}|j||||d} |j| | |||| d}|d}|jdk r<||nd}| sZ||f|d dSt|||j|jd S) NzDYou cannot specify both input_ids and inputs_embeds at the same timermz5You have to specify either input_ids or inputs_embeds)rr)rrlrr)rrrrrrr)rZ pooler_outputrr)rLrruse_return_dict ValueErrorZ%warn_if_padding_and_no_attention_maskrrrIr[rZrZget_extended_attention_maskZ get_head_maskrrrrr rr)r\rrrrlrrrrrrrZextended_attention_maskZembedding_outputZencoder_outputsrrr%r%r(rbKsV    zMobileBertModel.forward)T) NNNNNNNNN)rdrerfrrXrrrrMOBILEBERT_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrIrrrrrrbrhr%r%r^r(r )s@   r z MobileBert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. cseZdZddgZfddZddZddZdeee j d fd d Z e e d eeeddeejeejeejeejeejeejeejeejeejeejeejeeefd ddZZS)MobileBertForPreTrainingcls.predictions.decoder.weightcls.predictions.decoder.biascs,t|t||_t||_|dSrV)rWrXr r#rclsrrr^r%r(rXs   z!MobileBertForPreTraining.__init__cCs |jjjSrVr%rrrr%r%r(get_output_embeddingssz.MobileBertForPreTraining.get_output_embeddingscCs||jj_dSrVr&r\Z new_embeddigsr%r%r(set_output_embeddingssz.MobileBertForPreTraining.set_output_embeddingsNnew_num_tokensracs*|j|jjj|dd|jj_tj|dSNT)r+Z transposed)r+Z_get_resized_lm_headr%rrrWresize_token_embeddingsr\r+r^r%r(r.s  z0MobileBertForPreTraining.resize_token_embeddingsrrr) rrrrlrrlabelsnext_sentence_labelrrrrac  Cs| dk r | n|jj} |j||||||| | | d } | dd\} }|| |\}}d}|dk r|dk rt}||d|jj|d}||dd|d}||}| s||f| dd}|dk r|f|S|St|||| j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Returns: Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Nrrrlrrrrrr/rm)r r r rr) rLrr#r%rrrsrrr)r\rrrrlrrr1r2rrrrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrr%r%r(rbs<. z MobileBertForPreTraining.forward)N) NNNNNNNNNNN)rdrerf_tied_weights_keysrXr'r)rrDrrrr.rrrrrr!rIrrrrrbrhr%r%r^r(r"s@    r"z8MobileBert Model with a `language modeling` head on top.cseZdZddgZfddZddZddZdeee j d fd d Z e e d eeeeddddeejeejeejeejeejeejeejeeeeeeeeefd ddZZS)MobileBertForMaskedLMr#r$cs6t|t|dd|_t||_||_|dSNF)r)rWrXr r#rr%rLrrr^r%r(rXs   zMobileBertForMaskedLM.__init__cCs |jjjSrVr&rr%r%r(r'sz+MobileBertForMaskedLM.get_output_embeddingscCs||jj_dSrVr&r(r%r%r(r)sz+MobileBertForMaskedLM.set_output_embeddingsNr*cs*|j|jjj|dd|jj_tj|dSr,r-r/r^r%r(r.s  z-MobileBertForMaskedLM.resize_token_embeddingsrz'paris'g= ףp=?rrrexpected_output expected_loss rrrrlrrr1rrrrac  Cs| dk r | n|jj} |j|||||||| | d } | d} || } d}|dk rpt}|| d|jj|d}| s| f| dd}|dk r|f|S|St|| | j| j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nr3rrmr/r logitsrr) rLrr#r%rrrsrrr)r\rrrrlrrr1rrrrrrr6r5rr%r%r(rb$s6  zMobileBertForMaskedLM.forward)N) NNNNNNNNNN)rdrerfr8rXr'r)rrDrrrr.rrrrr rr!rIrrrrrrbrhr%r%r^r(r9 sH    r9cs0eZdZfddZejejdddZZS)MobileBertOnlyNSPHeadcstt|jd|_dSr)rWrXrrwrqrrr^r%r(rXbs zMobileBertOnlyNSPHead.__init__)rracCs||}|SrV)r)r\rrr%r%r(rbfs zMobileBertOnlyNSPHead.forwardrcr%r%r^r(rAas rAzPMobileBert Model with a `next sentence prediction (classification)` head on top.cseZdZfddZeedeee dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeeefd ddZZS) #MobileBertForNextSentencePredictioncs,t|t||_t||_|dSrV)rWrXr r#rAr%rrr^r%r(rXps   z,MobileBertForNextSentencePrediction.__init__rr0Nr>c  Ksd| krtdt| d}| dk r*| n|jj} |j|||||||| | d } | d} || }d}|dk rt}|| dd| d}| s|f| dd}|dk r|f|S|St ||| j | j dS) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Returns: Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```r2zoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nr3rrmr/r?) warningswarn FutureWarningpoprLrr#r%rrrrr)r\rrrrlrrr1rrrkwargsrrrr7r5rr%r%r(rbysB,   z+MobileBertForNextSentencePrediction.forward) NNNNNNNNNN)rdrerfrXrrrrrr!rrIrrrrrrbrhr%r%r^r(rBks4   rBz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZeedeee e e e dd e eje eje eje eje eje eje eje ee ee eeeeje fd ddZZS) #MobileBertForSequenceClassificationcsdt||j|_||_t||_|jdk r4|jn|j}t ||_ t |j |j|_ |dSrV)rWrX num_labelsrLr r#classifier_dropoutr|rr{r}rwrqr.rr\rLrJr^r%r(rXs   z,MobileBertForSequenceClassification.__init__rr;Nr>c  Cs|| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk r8|jjdkr|jdkrzd|j_n4|jdkr|jtj ks|jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n || |}nN|jjdkrt }|| d|j|d}n|jjdkr8t}|| |}| sh| f| dd}|dk rd|f|S|St|| | j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr3rZ regressionZsingle_label_classificationZmulti_label_classificationrmr/r?)rLrr#r}r.Z problem_typerIrrIrrDr squeezerrrrrr)r\rrrrlrrr1rrrrrr@r r5rr%r%r(rbsV       "     z+MobileBertForSequenceClassification.forward) NNNNNNNNNN)rdrerfrXrrrr'_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATIONrr!_SEQ_CLASS_EXPECTED_OUTPUT_SEQ_CLASS_EXPECTED_LOSSrrIrgrrrrbrhr%r%r^r(rHs@   rHz MobileBert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e e e e edd eejeejeejeejeejeejeejeejeeeeeeeeeje fd ddZZS) MobileBertForQuestionAnsweringcs@t||j|_t|dd|_t|j|j|_| dSr:) rWrXrIr r#rrwrq qa_outputsrrr^r%r(rX@s  z'MobileBertForQuestionAnswering.__init__r)rrrZqa_target_start_indexZqa_target_end_indexr<r=N) rrrrlrrstart_positions end_positionsrrrrac  CsP| dk r | n|jj} |j||||||| | | d } | d} || }|jddd\}}|d}|d}d}|dk r|dk rt|dkr|d}t|dkr|d}|d}| d|}| d|}t |d}|||}|||}||d}| s:||f| dd}|dk r6|f|S|St |||| j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nr3rrrmr)Z ignore_indexr/)r  start_logits end_logitsrr)rLrr#rQr=rLrrCrclamprrrr)r\rrrrlrrrRrSrrrrrr@rTrUr4Z ignored_indexr5Z start_lossZend_lossrr%r%r(rbJsP"           z&MobileBertForQuestionAnswering.forward) NNNNNNNNNNN)rdrerfrXrrrr_CHECKPOINT_FOR_QArr!_QA_TARGET_START_INDEX_QA_TARGET_END_INDEX_QA_EXPECTED_OUTPUT_QA_EXPECTED_LOSSrrIrgrrrrbrhr%r%r^r(rP7sH  rPz MobileBert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e e j e ee ee eeee j e fd ddZZS) MobileBertForMultipleChoicecsTt|t||_|jdk r&|jn|j}t||_t |j d|_ | dS)Nr) rWrXr r#rJr|rr{r}rwrqr.rrKr^r%r(rXs   z$MobileBertForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrNr>c  Cst| dk r | n|jj} |dk r&|jdn|jd} |dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|d|dnd}|j|||||||| | d } | d} || } || }|d| }d}|dk r0t}|||}| s`|f| dd}|dk r\|f|S|St ||| j | j dS)aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrmrr3r/r?) rLrrFrrr#r}r.rrrr)r\rrrrlrrr1rrrZ num_choicesrrr@Zreshaped_logitsr r5rr%r%r(rbsL      z#MobileBertForMultipleChoice.forward) NNNNNNNNNN)rdrerfrXrrrrr rr!rrIrgrrrrbrhr%r%r^r(r\s@ r\z MobileBert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeedeee e e e dd e eje eje eje eje eje eje eje ee ee eeeeje fd ddZZS) MobileBertForTokenClassificationcsbt||j|_t|dd|_|jdk r2|jn|j}t||_ t |j |j|_ | dSr:)rWrXrIr r#rJr|rr{r}rwrqr.rrKr^r%r(rX s  z)MobileBertForTokenClassification.__init__rr;Nr>c  Cs| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk rxt}|| d|j|d}| s| f| dd}|dk r|f|S|St|| | j | j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr3rrmr/r?) rLrr#r}r.rrrIrrr)r\rrrrlrrr1rrrrrr@r r5rr%r%r(rbs8   z(MobileBertForTokenClassification.forward) NNNNNNNNNN)rdrerfrXrrrr$_CHECKPOINT_FOR_TOKEN_CLASSIFICATIONrr!_TOKEN_CLASS_EXPECTED_OUTPUT_TOKEN_CLASS_EXPECTED_LOSSrrIrgrrrrbrhr%r%r^r(r]s@   r])]rr5rC dataclassesrtypingrrrrIrZtorch.nnrrr Z activationsr Zmodeling_outputsr r rrrrrrZmodeling_utilsrZ pytorch_utilsrrutilsrrrrrrZconfiguration_mobilebertrZ get_loggerrdr3r r!r^r_r`rWrZr[rXrYrMrNrOrrTModulerUr"ryrjrrrrrrrrrrrrrrrrrrrZMOBILEBERT_START_DOCSTRINGrr r"r9rArBrHrPr\r]r%r%r%r(s   (    N L:2 $  ?*  !2kkV ```Y