U ,-e;@s4dZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZmZmZmZmZmZdd lmZmZmZdd lm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)d dl*m+Z+e(,e-Z.dZ/dZ0ddddgZ1ddZ2d9ddZ3Gddde j4Z5Gddde j4Z6dZ7dZ8e&de7Gdd d e j4Z9Gd!d"d"eZ:Gd#d$d$e:Z;e&d%e7Gd&d'd'e:Ze&d.e7Gd/d0d0e:Z?e&d1e7eGd2d3d3e$Z@Gd4d5d5e:ZAe&d6e7Gd7d8d8e:ZBdS):z& PyTorch Flaubert model, based on XLM.N) dataclass)DictOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModelSequenceSummary SQuADHead)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)FlaubertConfigzflaubert/flaubert_base_casedr zflaubert/flaubert_small_casedzflaubert/flaubert_base_uncasedzflaubert/flaubert_large_casedc stfddt|D}tt|dddddf|dddddf<tt|dddddf|dddddf<|d|_dS)Ncs$g|]fddtDqS)c s(g|] }tdd|dqS)i')nppower).0j)dimposo/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/flaubert/modeling_flaubert.py Asz;create_sinusoidal_embeddings...)range)r$r&)r'r)r*Asz0create_sinusoidal_embeddings..rr!rF) r"arrayr+torch FloatTensorsincosZdetach_Z requires_grad)Zn_posr&outZ position_encr(r,r)create_sinusoidal_embeddings@s 44r3cCstj|tj|jd}|dk r"|}n(||ks6t||dddfk}|d}|r|ddddf||d|ddddfk}n|}|||fkst|dks||||fkst||fS)zH Generate hidden states mask, and optionally an attention mask. dtypedeviceNrrF) r.arangelongr6maxitemAssertionErrorsizerepeat)slenlengthscausal padding_maskalenmaskbs attn_maskr(r(r) get_masksIs 0rFcs6eZdZeZfddZddZd ddZZ S) MultiHeadAttentioncstttj|_||_||_|j|_ |j|jdks>t t |||_ t |||_t |||_t |||_t|_dS)Nr)super__init__nextrGNEW_IDlayer_idr&n_headsattention_dropoutdropoutr;rLinearq_link_linv_linout_linset pruned_heads)selfrMr&config __class__r(r)rIfs  zMultiHeadAttention.__init__cCs|j|j}t|dkrdSt||j||j\}}t|j||_t|j||_t|j||_t|j |dd|_ |jt||_||j|_|j ||_dS)Nrrr,) r&rMlenrrVrrQrRrSrTunion)rWheadsZattention_head_sizeindexr(r(r) prune_headsts   zMultiHeadAttention.prune_headsNFcs*|\}}|dkr0|dkr"|n |d|} n |d} j} j| |dkrbd|| fn dd| f} fdd} fdd} | |}|dkr| |}| |}n6|dksԈj|kr|}}| |}| |}|dk rhj|krZ|dkrL|j\}}tj||gd d }tj||gd d }n|j\}}||f|j<|t }t || d d}|d k | |}||t|jjtjj|d d |}tjj|jjd }|dk r||}t ||}| |}|f}|r&||f}|S)zd Self-attention (if kv is None) or attention over source sentence (provided by kv). Nr>rr cs|djddS)Z projectionrr!)viewrM transposexrDZ dim_per_headrWr(r)shapesz)MultiHeadAttention.forward..shapecs |dddjS)zcompute contextrr!r`)rb contiguousrarMrcrer(r)unshapesz+MultiHeadAttention.forward..unshaper!r,rr`ptraining)r<rMr&rQrRrSrLr.catmathsqrtmatmulrbra expand_asZ masked_fill_Zfinfor5minr functionalZsoftmaxfloatZtype_asrOrkrT)rWinputrCkvcache head_maskoutput_attentionsqlenr&ZklenrMZ mask_reshaperfrhqkvZk_Zv_scoresweightscontextoutputsr(rer)forwardsN  $       zMultiHeadAttention.forward)NNNF) __name__ __module__ __qualname__ itertoolscountrKrIr_r __classcell__r(r(rYr)rGcs rGcs,eZdZfddZddZddZZS)TransformerFFNcsTt|j|_t|||_t|||_|jr8tntj j |_ |j |_ d|_ dSNr)rHrIrOrrPlin1lin2Zgelu_activationr rrZreluactchunk_size_feed_forward seq_len_dim)rWZin_dimZ dim_hiddenZout_dimrXrYr(r)rIs zTransformerFFN.__init__cCst|j|j|j|SN)rff_chunkrr)rWrtr(r(r)rszTransformerFFN.forwardcCs8||}||}||}tjj||j|jd}|S)Nri)rrrrrrrOrk)rWrtrdr(r(r)rs    zTransformerFFN.ff_chunk)rrrrIrrrr(r(rYr)rs raB This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`FlaubertConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) lengths (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Length of each sentence that can be used to avoid performing attention on padding token indices. You can also use `attention_mask` for the same result (see above), kept here for compatibility. Indices selected in `[0, ..., input_ids.size(-1)]`: cache (`Dict[str, torch.FloatTensor]`, *optional*): Dictionary strings to `torch.FloatTensor` that contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see `cache` output below). Can be used to speed up sequential decoding. The dictionary object will be modified in-place during the forward pass to add newly computed hidden-states. head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. zbThe bare Flaubert Model transformer outputting raw hidden-states without any specific head on top.cs*eZdZdZfddZdddZZS)FlaubertPredLayerz? Prediction layer (cross_entropy or adaptive_softmax). csht|j|_|j|_|j|_|j}|jdkrHtj||jdd|_ntj ||j|j |j dd|_dS)NFT)bias)Z in_featuresZ n_classesZcutoffsZ div_valueZ head_bias) rHrIasmn_words pad_indexemb_dimrrPprojZAdaptiveLogSoftmaxWithLossZ asm_cutoffsZ asm_div_value)rWrXr&rYr(r)rI.s  zFlaubertPredLayer.__init__NcCsd}|jdkrX||}|f|}|dk rtjj|d|j|ddd}|f|}n8|j|}|f|}|dk r|||\}}|f|}|S)z,Compute the loss, and optionally the scores.r(FNr`mean)Z reduction)rrrrrZ cross_entropyrarZlog_prob)rWrdyrr}loss_r(r(r)r@s   "    zFlaubertPredLayer.forward)N)rrr__doc__rIrrr(r(rYr)r$s rcs@eZdZdZeZdZdZfddZe ddZ dd Z Z S) FlaubertPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. N transformercstj||dSr)rHrI)rWinputskwargsrYr(r)rI^sz FlaubertPreTrainedModel.__init__c Cstdddddgdddddgdddddgg}tdddddgdddddgdddddgg}|jjr|jjdkrtdddddgdddddgdddddgg}nd}|||d S) Nrrr!r ) input_idsattention_masklangs)r.tensorrX use_lang_embn_langs)rWZ inputs_listZ attns_listZ langs_listr(r(r) dummy_inputsas ..0z$FlaubertPreTrainedModel.dummy_inputscCst|tjrV|jdk r:|jjdk r:tjj|jd|jjd|jdk rV|jj |j t|tj r|jdk r|jj dk rtjj|jd|jj d|j dk rtj|j dt|tjr|j j |jj ddS)zInitialize the weights.Nr)rZstdg?) isinstancer EmbeddingrXZembed_init_stdinitZnormal_weight padding_idxdataZzero_rPZinit_stdrZ constant_ LayerNormZfill_)rWmoduler(r(r) _init_weightsks      z%FlaubertPreTrainedModel._init_weights) rrrrr config_classZload_tf_weightsZbase_model_prefixrIpropertyrrrr(r(rYr)rTs  rcseZdZfddZddZddZddZeee e e e d de eje eje eje eje eje eje eeejfe eje eje ee ee eeee fd d d ZZS) FlaubertModelc stt||j|_|j |_|jr,td|j|_|j|_|j|_|j|_|j |_ |j |_ |j |_ |j d|_ |j|_|j|_|j|_|j|_|j |jdkstdt|j|j |_|jrt|j|j |jjd|jdkr|jrt|j|j |_tj|j|j |j d|_tj|j |jd|_t|_ t|_!t|_"t|_#t$|jD]r}|j %t&|j|j |d |j!%tj|j |jd|j"%t'|j |j |j |d |j#%tj|j |jdqVt(|d r.|j)*+}i|_)|D]>\}}|j t,|j|jkr|-t,|t.t/t,|iq|0t1|d d |_2t1|d d|_3|j4dt56|j7ddddS)Nz1Currently Flaubert can only be used as an encoderrrz-transformer dim must be a multiple of n_heads)r2r)r)Zeps)rXrV layerdroprpre_normF position_ids)rr`) persistent)8rHrIZ is_encoderZ is_decoderNotImplementedErrorr@rrrZ eos_indexrrr&Z hidden_dimrMn_layersrOrNr;rrZmax_position_embeddingsposition_embeddingsZsinusoidal_embeddingsr3rlang_embeddings embeddingsrZlayer_norm_epslayer_norm_embZ ModuleList attentions layer_norm1ffns layer_norm2r+appendrGrhasattrrVcopyitemsintr_listmap post_initgetattrrrZregister_bufferr.r7expand)rWrXrrVlayerr]rYr(r)rI}s`          zFlaubertModel.__init__cCs|jSrrrWr(r(r)get_input_embeddingssz"FlaubertModel.get_input_embeddingscCs ||_dSrrrWZnew_embeddingsr(r(r)set_input_embeddingssz"FlaubertModel.set_input_embeddingscCs&|D]\}}|j||qdS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)rrr_)rWZheads_to_prunerr]r(r(r) _prune_headsszFlaubertModel._prune_heads checkpoint output_typerN) rrrtoken_type_idsrr?rvrw inputs_embedsrxoutput_hidden_states return_dictreturnc Cs| dk r | n|jj} | dk r | n|jj} | dk r4| n|jj} |dk rR|\} }n| dd\} }|dk rt|jn| j}|dkr|dk r||jkjdd}nt j |g| |d}|d| kst | |kst t|||j|d\}}|dkrVt|dr.|jddd|f}|| |f}n&t j|t j|d}|d| |f}n|| |fkslt |dk r|| |fkst |||jj}|dk r4|dk r4||d }|dd| df}|dd| df}|dk r|dd| df}|dd| df}|dd| df}| dkrH||} | ||| }|dk r|jr|jjdkr|||}|dk r|||}||}tjj ||j |j!d }||d"|j#9}| rd nd}| rd nd}t$|jD]h}|j!r&t %g}||j&kr&q| r6||f}|j's|j(||||||| d }|d}| rv||df}tjj ||j |j!d }||}|j)||}nb|j)||}|j(||||||d }|d}| r||df}tjj ||j |j!d }||}|j's0||j*||}|j+||}n |j+||}||j*||}||d"|j#9}q| rz||f}|dk r|d |d7<| st,dd|||fDSt-|||dS)Nr`rr,)r6r)rArr4r>rir()rvrwrx)rvrwcss|]}|dk r|VqdSrr()r$r|r(r(r) sz(FlaubertModel.forward..)Zlast_hidden_state hidden_statesr).rXrxruse_return_dictr<r6rsumr8r.rr;r9r:rFr@rrrr7Z unsqueezeZ get_head_maskrrrrprrrrrrrrOrktor5r+Zrandrrrrrrtupler )rWrrrrrr?rvrwrrxrrrDr>r6rCrEZ_slenrrriZdropout_probabilityZ attn_outputsZattnZtensor_normalizedr(r(r)rs                zFlaubertModel.forward) NNNNNNNNNNNN)rrrrIrrrrFLAUBERT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrr. LongTensorr/Tensorrstrboolrrrrr(r(rYr)r|sJ L rz The Flaubert Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). cseZdZdgZfddZddZddZdd Zee d e e e ed d deejeejeejeejeejeejeeeejfeejeejeejeeeeeeeee fdddZZS)FlaubertWithLMHeadModelzpred_layer.proj.weightcs,t|t||_t||_|dSr)rHrIrrr pred_layerrrWrXrYr(r)rIs   z FlaubertWithLMHeadModel.__init__cCs|jjSrrrrr(r(r)get_output_embeddingssz-FlaubertWithLMHeadModel.get_output_embeddingscCs ||j_dSrrrr(r(r)set_output_embeddingssz-FlaubertWithLMHeadModel.set_output_embeddingscKsj|jj}|jj}|jd}tj|df|tj|jd}tj||gdd}|dk r\t ||}nd}||dS)Nrrr4r,)rr) rX mask_token_idlang_idrfr.fullr8r6rlZ full_like)rWrrrrZeffective_batch_sizeZ mask_tokenrr(r(r)prepare_inputs_for_generations z5FlaubertWithLMHeadModel.prepare_inputs_for_generationbatch_size, sequence_lengthz )rrrrCNrrrrrr?rvrwrlabelsrxrrrcCs| dk r | n|jj} |j||||||||| | | | d }|d}||| }| s^||ddSt| dk rp|dnd| dkr|dn|d|j|jdS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N rrrrr?rvrwrrxrrrrrlogitsrr)rXrrrrrr)rWrrrrrr?rvrwrrrxrrtransformer_outputsoutputrr(r(r)rs2 zFlaubertWithLMHeadModel.forward) NNNNNNNNNNNNN)rrrZ_tied_weights_keysrIrrrrrformatrrrrrr.rrrrrrrrr(r(rYr)rsR    rz Flaubert Model with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e eee j fe e j e e j e e j e ee ee eeee fdddZZS) !FlaubertForSequenceClassificationcs:t||j|_||_t||_t||_|dSr) rHrI num_labelsrXrrrsequence_summaryrrrYr(r)rIs    z*FlaubertForSequenceClassification.__init__rrNrcCsx| dk r | n|jj} |j||||||||| | | | d }|d}||}d}| dk r4|jjdkr|jdkrvd|j_n4|jdkr| jtjks| jtj krd|j_nd|j_|jjdkrt }|jdkr|| | }n ||| }nN|jjdkrt }|| d|j| d}n|jjdkr4t}||| }| sd|f|dd}|dk r`|f|S|St|||j|jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrZ regressionZsingle_label_classificationZmulti_label_classificationr`r)rXrrrZ problem_typerr5r.r8rr squeezer rarrrr)rWrrrrrr?rvrwrrrxrrrrrrloss_fctr(r(r)rsZ     "     z)FlaubertForSequenceClassification.forward) NNNNNNNNNNNNN)rrrrIrrrrrrrrr.rrrrrrrrr(r(rYr)rsH  rz Flaubert Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e eee j fe e j e e j e e j e ee ee eeee fdddZZS) FlaubertForTokenClassificationcsJt||j|_t||_t|j|_t|j |j|_ | dSr) rHrIrrrrZDropoutrOrP hidden_size classifierrrrYr(r)rIWs   z'FlaubertForTokenClassification.__init__rrNrcCs| dk r | n|jj} |j||||||||| | | | d }|d}||}||}d}| dk r~t}||d|j| d}| s|f|dd}|dk r|f|S|St|||j |j dS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr`rr) rXrrrOrr rarrrr)rWrrrrrr?rvrwrrrxrrrsequence_outputrrrrr(r(r)rbs>  z&FlaubertForTokenClassification.forward) NNNNNNNNNNNNN)rrrrIrrrrrrrrr.rrrrrrrrr(r(rYr)rNsH  rz Flaubert Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e eee j fe e j e e j e e j e e j e ee ee eeee fdddZZS) "FlaubertForQuestionAnsweringSimplecs4t|t||_t|j|j|_| dSr) rHrIrrrrPrr qa_outputsrrrYr(r)rIs  z+FlaubertForQuestionAnsweringSimple.__init__rrN)rrrrrr?rvrwrstart_positions end_positionsrxrrrcCsV|dk r |n|jj}|j||||||||| | | |d }|d}||}|jddd\}}|d}|d}d}| dk r| dk rt| dkr| d} t| dkr| d} |d}| d|} | d|} t |d}||| }||| }||d}|s@||f|dd}|dk r<|f|S|St ||||j |j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. Nrrrr`r,)Z ignore_indexr!)r start_logits end_logitsrr)rXrrr splitrrgr[r<clampr rrr)rWrrrrrr?rvrwrr r rxrrrr rrrZ total_lossZ ignored_indexrZ start_lossZend_lossrr(r(r)rsV!          z*FlaubertForQuestionAnsweringSimple.forward)NNNNNNNNNNNNNN)rrrrIrrrrrrrrr.rrrrrrrrr(r(rYr)r sL  r z Flaubert Model with a beam-search span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). c@seZdZUdZdZeejed<dZ eejed<dZ eej ed<dZ eejed<dZ eej ed<dZeejed<dZeeejed <dZeeejed <dS) "FlaubertForQuestionAnsweringOutputa9 Base class for outputs of question answering models using a `SquadHead`. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nrstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logitsrr)rrrrrrr.r/__annotations__rrrrrrrrrr(r(r(r)r s rcseZdZfddZeedeee dd e e j e e j e e j e e j e e j e e j e e ee j fe e j e e j e e j e e j e e j e e j e e j e ee ee eeeefdddZZS) FlaubertForQuestionAnsweringcs,t|t||_t||_|dSr)rHrIrrrr rrrYr(r)rIAs   z%FlaubertForQuestionAnswering.__init__r)rrN)rrrrrr?rvrwrr r  is_impossible cls_indexp_maskrxrrrcCs|dk r |n|jj}|j||||||||| |||d }|d}|j|| | | | ||d}|sj||ddSt|j|j|j|j|j |j |j |j dS)a6 start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. is_impossible (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels whether a question has an answer or no answer (SQuAD 2.0) cls_index (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the classification token to use as input for computing plausibility of the answer. p_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Optional mask of tokens which can't be in answers (e.g. [CLS], [PAD], ...). 1.0 means token should be masked. 0.0 mean token is not masked. Returns: Example: ```python >>> from transformers import XLMTokenizer, XLMForQuestionAnswering >>> import torch >>> tokenizer = XLMTokenizer.from_pretrained("xlm-mlm-en-2048") >>> model = XLMForQuestionAnswering.from_pretrained("xlm-mlm-en-2048") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze( ... 0 ... ) # Batch size 1 >>> start_positions = torch.tensor([1]) >>> end_positions = torch.tensor([3]) >>> outputs = model(input_ids, start_positions=start_positions, end_positions=end_positions) >>> loss = outputs.loss ```Nrr)r r rrrrr)rrrrrrrr) rXrrr rrrrrrrrr)rWrrrrrr?rvrwrr r rrrrxrrrrrr(r(r)rJsJ< z$FlaubertForQuestionAnswering.forward)NNNNNNNNNNNNNNNNN)rrrrIrrrrrrrr.rrrrrrrrr(r(rYr)r@sP   rz Flaubert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a softmax) e.g. for RocStories/SWAG tasks. cseZdZfddZeedeee e dd e e j e e j e e j e e j e e j e e j e eee j fe e j e e j e e j e ee ee eeee fdddZZS) FlaubertForMultipleChoicecsDtj|f||t||_t||_t|jd|_ | dSr) rHrIrrrrrrPr logits_projr)rWrXrrrYr(r)rIs   z"FlaubertForMultipleChoice.__init__z(batch_size, num_choices, sequence_lengthrNrcCs| dk r | n|jj} |dk r&|jdn| jd}|dk rJ|d|dnd}|dk rh|d|dnd}|dk r|d|dnd}|dk r|d|dnd}|dk r|d|dnd}| dk r| d| d| dnd} |dk rtdd}|j||||||||| | | | d }|d}||}| |}|d|}d}| dk rlt }||| }| s|f|dd}|dk r|f|S|St |||j |j dS) aJ labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr`zwThe `lengths` parameter cannot be used with the Flaubert multiple choice models. Please use the attention mask instead.) rrrrrr?rvrwrrxrrrr)rXrrfrar<loggerwarningrrrr rrr)rWrrrrrr?rvrwrrrxrrZ num_choicesrrrZreshaped_logitsrrr(r(r)rs^      z!FlaubertForMultipleChoice.forward) NNNNNNNNNNNNN)rrrrIrrrrrrrrr.rrrrrrrrr(r(rYr)rsL  r)N)Crrrm dataclassesrtypingrrrrnumpyr"r.rZtorch.nnrr r Z activationsr Zmodeling_outputsr rrrrrZmodeling_utilsrrrZ pytorch_utilsrrrutilsrrrrrrZconfiguration_flaubertr Z get_loggerrr rrZ&FLAUBERT_PRETRAINED_MODEL_ARCHIVE_LISTr3rFModulerGrZFLAUBERT_START_DOCSTRINGrrrrrrrr rrrr(r(r(r)s         a;+( Z_Ma+r