U ,-e@sLdZddlZddlZddlmZmZmZmZddlZddl m m Z ddl Zddlm Z ddlmZmZmZddlmZddlmZmZmZdd lmZdd lmZmZmZmZmZd d l m!Z!erdd l"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(de)e*e#j+kZ,e-e.Z/dZ0ddZ1d9ej2ej3ej4e5e5dddZ6d:ej7ej3ee5dddZ8Gddde j9Z:Gddde j9Z;ddZej7e5ej7d$d%d&Z?Gd'd(d(e j9Z@Gd)d*d*e@ZAGd+d,d,e j9ZBd-ZCed.eCGd/d0d0eZDd1ZEed.eCGd2d3d3eDZFGd4d5d5eDZGed6eCGd7d8d8eDZHdS);z PyTorch Mistral model.N)ListOptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)PreTrainedModel)add_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_availableloggingreplace_return_docstrings) MistralConfig)flash_attn_funcflash_attn_varlen_func)index_first_axis pad_input unpad_input window_sizercCsX|jdtjd}tj|dd}|}ttj |dtjjdd}|||fS)NdimdtypeF)as_tupler)rr) sumtorchint32ZnonzeroflattenmaxitemFpadcumsum) padding_maskZseqlens_in_batchindicesZmax_seqlen_in_batchZ cu_seqlensr-m/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/mistral/modeling_mistral.py_get_unpad_data9s r/)input_ids_shaper devicepast_key_values_lengthsliding_windowc Cs|\}}tj||fd|d}tj|dd}tj|| d}t||}|dkrrtjtj||||d|gdd}|ddddddf|d|||S) z< Make causal mask used for sliding window attention r)Z fill_valuer2r)Zdiagonalr r2rrN) r#fullZtrilZtriulogtocatZzerosexpand) r1r r2r3r4bsztgt_lentensormaskr-r-r. _make_sliding_window_causal_maskEs  r@)r?r r=cCsj|\}}|dk r|n|}|ddddddf|d|||}d|}||tjt|jS)z_ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. Nr?)sizer;r9Z masked_fillr#boolZfinfomin)r?r r=r<Zsrc_lenZ expanded_maskZ inverted_maskr-r-r. _expand_maskas  *rEcs&eZdZdfdd ZddZZS)MistralRMSNormư>cs&ttt||_||_dS)z= MistralRMSNorm is equivalent to T5LayerNorm N)super__init__r Parameterr#onesweightvariance_epsilon)self hidden_sizeeps __class__r-r.rIqs zMistralRMSNorm.__init__cCsJ|j}|tj}|djddd}|t||j}|j||S)NrT)Zkeepdim) r r9r#float32powmeanZrsqrtrMrL)rN hidden_states input_dtypeZvariancer-r-r.forwardys  zMistralRMSNorm.forward)rG__name__ __module__ __qualname__rIrY __classcell__r-r-rQr.rFpsrFcs0eZdZd fdd ZddZd dd ZZS) MistralRotaryEmbedding'Ncsrt||_||_||_d|jtd|jd||j}|j d|dd|j ||j j t ddS)NrArrSinv_freqF persistentseq_lenr2r )rHrIrmax_position_embeddingsbaser#arangefloatr9register_buffer_set_cos_sin_cacherbr2Zget_default_dtype)rNrrgrhr2rbrQr-r.rIs *zMistralRotaryEmbedding.__init__cCs||_tj|j||jjd}td||j}tj||fdd}|jd|ddddddf |dd|jd| ddddddf |dddS) N)r2r zi,j->ijrr6 cos_cachedFrc sin_cached) max_seq_len_cachedr#rirbr Zeinsumr:rkcosr9sin)rNrfr2r tZfreqsZembr-r-r.rls .z)MistralRotaryEmbedding._set_cos_sin_cachecCsn||jkr|j||j|jd|jddddd|dfj|jd|jddddd|dfj|jdfS)Nre.)r )rorlr2r rmr9rn)rNxrfr-r-r.rYs  &&zMistralRotaryEmbedding.forward)r`raN)N)r[r\r]rIrlrYr^r-r-rQr.r_s r_cCsH|dd|jddf}|d|jdddf}tj| |fddS)z*Rotates half the hidden dims of the input..NrrSr6)shaper#r:)rsx1Zx2r-r-r. rotate_halfsrvcCsl|dd}|dd}||d}||d}||t||}||t||}||fS)Nrr)squeeze unsqueezerv)qkrprq position_idsZq_embedZk_embedr-r-r.apply_rotary_pos_embsr|cs$eZdZfddZddZZS) MistralMLPcsrt||_|j|_|j|_tj|j|jdd|_tj|j|jdd|_tj|j|jdd|_ t |j |_ dSNFbias) rHrIconfigrOZintermediate_sizerLinear gate_projup_proj down_projr Z hidden_actact_fnrNrrQr-r.rIs zMistralMLP.__init__cCs ||||||SN)rrrr)rNrsr-r-r.rYszMistralMLP.forwardrZr-r-rQr.r}s r})rWn_repreturncCs^|j\}}}}|dkr|S|dddddddddf|||||}||||||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rtr;reshape)rWrbatchnum_key_value_headsslenhead_dimr-r-r. repeat_kvs 0rcseZdZdZedfdd ZejeedddZ d eje eje ej e e eje e e eje eje eje e ejfd d d ZZS)MistralAttentionz Multi-headed attention from 'Attention Is All You Need' paper. Modified to use sliding window attention: Longformer and "Generating Long Sequences with Sparse Transformers". rcs t||_|j|_|j|_|j|j|_|j|_|j|j|_|j |_ |j |_ |j|j|jkrt d|jd|jdt j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t j |j|j|jdd|_t|j|j |j d|_dS)Nz?hidden_size must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).Fr)rgrh)rHrIrrOZnum_attention_heads num_headsrrnum_key_value_groupsrgZ rope_theta ValueErrorrrq_projk_projv_projo_projr_ rotary_embrrQr-r.rIs, zMistralAttention.__init__)r>rfr<cCs ||||j|jddS)NrrS)viewrr transpose contiguous)rNr>rfr<r-r-r._shapeszMistralAttention._shapeNFrWattention_maskr{past_key_valueoutput_attentions use_cacher+rcCsn|\}} } ||} ||} ||} | || |j|jdd} | || |j|jdd} | || |j|jdd} | j d}|dk r||dj d7}|j | |d\}}t | | |||\} } |dk rt j |d| gdd} t j |d| gdd} |r | | fnd}t| |j} t| |j} t | | ddt|j}|||j| |fkrtd||j| |fd ||dk r||d| |fkrtd |d| |fd |||}tjj|d t jd | j}t || }|||j| |jfkr0td ||j| |jfd ||dd}||| |j}||}|sdd}|||fS)NrrSrrfr6r z$Attention weights should be of size z , but is z!Attention mask should be of size rrz `attn_output` should be of size )rBrrrrrrrrrtrr|r#r:rrmatmulmathsqrtrr functionalZsoftmaxrTr9r rrrOr)rNrWrr{rrrr+r<q_len_ query_states key_states value_states kv_seq_lenrprq attn_weights attn_outputr-r-r.rYsR           zMistralAttention.forward)NNNFFN)r[r\r]__doc__rrIr#Tensorintrr LongTensorrrCrYr^r-r-rQr.rs& rc @s\eZdZdZd ejeejeejeeeje e eejdddZ d dd Z d d Z dS)MistralFlashAttention2aJ Mistral flash attention module. This module inherits from `MistralAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. NFrWrr{rrrr+c Cs(|\}} } ||} ||} ||} | || |j|jdd} | || |j|jdd} | || |j|jdd} | j d}|dk r||dj d7}t ||dddf d}|j | |d\}}t | | |||\} } tot|jddk o||jjk}ts td|dk rLt|jdr ||jjkr ||jj}|d}|d}|dddd|dddf}|dddd|dddf}|j d|jjdkrtd |j ||f}|dk r |dd|df}tj|t|ddddfgdd }tj|d| gdd } tj|d| gdd } |rZ| | fnd}t| |j} t| |j} d }| j}|tjkrtd | tj} | tj} | tj} | dd} | dd} | dd} |j| | | || ||d }| || |j!}|"|}|sd}|||fS)NrrSrrrrr4zThe current flash attention version does not support sliding window attention, for a more memory efficient implementation make sure to upgrade flash-attn library.zepast key much have a shape of (`batch_size, num_heads, self.config.sliding_window-1, head_dim`), got r6zThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in float16.)dropoutuse_sliding_windows)#rBrrrrrrrrrtr&r'rr|_flash_supports_window_sizehasattrrr4logger warning_oncerrr#r:Z ones_likerrr rTr9Zfloat16_flash_attention_forwardrrOr)rNrWrr{rrrr+r<rrrrrrZrotary_seq_lenrprqrZslicing_tokensZpast_keyZ past_valueZ dropout_raterXrrr-r-r.rYJs     "   $$  (           zMistralFlashAttention2.forwardrc  Cs|dk r|jd} ||||||\}}}} } } | \} }| \}}|sbt|||| |||||dd }n*t|||| |||||d|jj|jjfd }t|| | |}n<|st|||||dd}n"t|||||d|jj|jjfd}|S)a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API padding_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`int`, *optional*): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) use_sliding_windows (`bool`, *optional*): Whether to activate sliding window attention. NrT) cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_k dropout_p softmax_scalecausal)rrrrrrrr)rr)rrr)rt _upad_inputrrr4rr)rNrrrr+ query_lengthrrr batch_size indices_qZ cu_seq_lensZ max_seq_lensrrmax_seqlen_in_batch_qmax_seqlen_in_batch_kZattn_output_unpadrr-r-r.rsr!   z/MistralFlashAttention2._flash_attention_forwardcCs&|j\}}}} ||jdkr>|jd} |dd| |df}t|\} } } t||||| | }t||||| | }||krt||||| | }| }| }| }nd|dkrd}tj|dtj|jd}|dd}|d}n(|dd| df}t ||\}}}}|||||| f|| ffS)Nrrr5) rtr/rrr#rir$r2rwr)rNZ query_layerZ key_layerZ value_layerr+rrrrrZpadding_mask_num_tokensZ indices_krrrrrr-r-r.rsB   z"MistralFlashAttention2._upad_input)NNNFFN)rNF) r[r\r]rr#rrrrrCrYrrr-r-r-r.rCs*   | YrcseZdZedfdd Zd ejeejeejee ejee ee eeje ej ee ej ej ffdddZ Z S) MistralDecoderLayerrcsdt|j|_t|dds(t|dnt||_t||_t |j|j d|_ t |j|j d|_ dS)N_flash_attn_2_enabledFrrP) rHrIrOgetattrrr self_attnr}mlprF rms_norm_epsinput_layernormpost_attention_layernormrrQr-r.rIDs   zMistralDecoderLayer.__init__NFrc Csz|}||}|j|||||||d\}} } ||}|}||}||}||}|f} |rh| | f7} |rv| | f7} | S)a] Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states r)rrrr) rNrWrr{rrrr+ZresidualZself_attn_weightsZpresent_key_valueoutputsr-r-r.rYPs,      zMistralDecoderLayer.forward)NNNFFN)r[r\r]rrIr#rrrrrC FloatTensorrYr^r-r-rQr.rCs" raL This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MistralConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. zUThe bare Mistral Model outputting raw hidden-states without any specific head on top.c@s8eZdZeZdZdZdgZdZdZ ddZ d dd Z d S) MistralPreTrainedModelmodelTrpast_key_valuescCs||jj}t|tjr>|jjjd|d|jdk rx|jj n:t|tj rx|jjjd|d|j dk rx|jj|j  dS)Nr)rVstd) rZinitializer_range isinstancerrrLdataZnormal_rZzero_ Embedding padding_idx)rNmodulerr-r-r. _init_weightss    z$MistralPreTrainedModel._init_weightsFcCst|tr||_dSr)r MistralModelgradient_checkpointing)rNrvaluer-r-r._set_gradient_checkpointings z2MistralPreTrainedModel._set_gradient_checkpointingN)F) r[r\r]r config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_skip_keys_device_placementZ_supports_flash_attn_2rrr-r-r-r.rs raV Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). If you want to change padding behavior, you should read [`modeling_opt._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more information on the default strategy. - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cseZdZdZedfdd ZddZddZd d Ze e de j e e je e j e ee je e je ee ee ee eeeefd d dZZS)rz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MistralDecoderLayer`] Args: config: MistralConfig rcsvtj|_j|_tjj|j|_t fddt j D|_ t jjd|_d|_|dS)Ncsg|] }tqSr-)r).0rrr-r. sz)MistralModel.__init__..rF)rHrI pad_token_idr vocab_sizerrrO embed_tokensZ ModuleListrangeZnum_hidden_layerslayersrFrnormr post_initrrQrr.rIs  zMistralModel.__init__cCs|jSrrrNr-r-r.get_input_embeddingssz!MistralModel.get_input_embeddingscCs ||_dSrrrNrr-r-r.set_input_embeddingssz!MistralModel.set_input_embeddingscCsbd}|ddkr&t||j|j||d}|dk r^t||j|dd|j}|dkrV|n||}|S)Nrr)r2r3r4)r=)r@r r2rEr9)rNrZ input_shape inputs_embedsr3r4Zcombined_attention_maskZexpanded_attn_maskr-r-r._prepare_decoder_attention_masks  z,MistralModel._prepare_decoder_attention_maskN) input_idsrr{rrrroutput_hidden_states return_dictrc  sBdk r n|jj|dk r |n|jj}|dk r4|n|jj}| dk rH| n|jj} |dk rj|dk rjtdn2|dk r~|j\} } n|dk r|j\} } } ntd| } d}|dk r|ddjd}| |} |dkr|dk r|jn|j}tj || |tj |d}| d d| }n| d|  }|dkr4| |}d|dkr\tj| | ftj|jd}nd|krj|dk rt|jdr|jjrdddf| k}|rtd|j|| | f|||jjd }|}|jr|jr|rtd d }|rd nd}rd nd}|r d nd}t|jD]\}}|rF||f7}|dk rX||nd|jr|jrfd d}tjj|||||}n|||||d}|d}|r||rdndf7}r.||df7}q.||}|r||f7}|r|nd}| s2tdd||||fDSt ||||dS)NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timezEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrSr5rrzYou are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to call `tokenizer.padding_side = 'left'` before tokenizing the input. )r4zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fr-csfdd}|S)Ncs|fdiS)Nr+r-)inputs)rrr+rr-r.custom_forwardszKMistralModel.forward..create_custom_forward..custom_forwardr-)rrrr+r)rr.create_custom_forwardsz3MistralModel.forward..create_custom_forward)rr{rrrr+rcss|]}|dk r|VqdSrr-)rvr-r-r. sz'MistralModel.forward..)Zlast_hidden_staterrW attentions)!rrrruse_return_dictrrtr2r#rilongrxrrrKrCrrr"r'rr4rZtrainingrr enumeraterutils checkpointrtupler )rNrrr{rrrrrrrZ seq_lengthrZseq_length_with_pastr3r2Zis_padding_rightrWZall_hidden_statesZall_self_attnsZnext_decoder_cacheidxZ decoder_layerrZ layer_outputsZ next_cacher-rr.rY/s             zMistralModel.forward) NNNNNNNNN)r[r\r]rrrIrrrrMISTRAL_INPUTS_DOCSTRINGr#rrrrrrCrrr rYr^r-r-rQr.rs6   rcseZdZdgZfddZddZddZdd Zd d Zd d Z ddZ e e e eeddejeejeejeeejeejeejeeeeeeeeeeefd ddZdddZeddZZS)MistralForCausalLMzlm_head.weightcs@t|t||_|j|_tj|j|jdd|_| dSr~) rHrIrrrrrrOlm_headrrrQr-r.rIs   zMistralForCausalLM.__init__cCs|jjSrrrrr-r-r.rsz'MistralForCausalLM.get_input_embeddingscCs ||j_dSrr rr-r-r.rsz'MistralForCausalLM.set_input_embeddingscCs|jSrr rr-r-r.get_output_embeddingssz(MistralForCausalLM.get_output_embeddingscCs ||_dSrr )rNZnew_embeddingsr-r-r.set_output_embeddingssz(MistralForCausalLM.set_output_embeddingscCs ||_dSrr)rNdecoderr-r-r. set_decoderszMistralForCausalLM.set_decodercCs|jSrrrr-r-r. get_decoderszMistralForCausalLM.get_decoder) output_typerN rrr{rrlabelsrrrrrc  Cs*|dk r |n|jj}| dk r | n|jj} | dk r4| n|jj} |j|||||||| | d } | d} || } | } d}|dk r| dddddf}|dddf}t}| d|jj }| d}| |j }|||}| s| f| dd}|dk r|f|S|St || | j| j| jdS)a Args: labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: Example: ```python >>> from transformers import AutoTokenizer, MistralForCausalLM >>> model = MistralForCausalLM.from_pretrained(PATH_TO_CONVERTED_WEIGHTS) >>> tokenizer = AutoTokenizer.from_pretrained(PATH_TO_CONVERTED_TOKENIZER) >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrr{rrrrrrr.rrlosslogitsrrWr)rrrrrr rjrrrrr9r2r rrWr)rNrrr{rrrrrrrrrWrrZ shift_logitsZ shift_labelsloss_fctoutputr-r-r.rYsJ)     zMistralForCausalLM.forwardcKs|r|ddddf}|dd}|dk rp|dkrp|dd}||dkd|rp|dddfd}|dk r|dkrd|i}nd|i}||||d|d|S) Nrr{rrrrr)r{rrr)getrr*Z masked_fill_rxupdate)rNrrrrkwargsr{Z model_inputsr-r-r.prepare_inputs_for_generation>s&  z0MistralForCausalLM.prepare_inputs_for_generationcs.d}|D] }|tfdd|Df7}q|S)Nr-c3s"|]}|d|jVqdS)rN)Z index_selectr9r2)rZ past_statebeam_idxr-r.rasz4MistralForCausalLM._reorder_cache..)r)rr Zreordered_pastZ layer_pastr-rr._reorder_cache\s z!MistralForCausalLM._reorder_cache) NNNNNNNNNN)NNN)r[r\r]Z_tied_weights_keysrIrrr rrrrrrr _CONFIG_FOR_DOCr#rrrrrrCrrrYr staticmethodr!r^r-r-rQr.r sN    X r a The Mistral Model transformer with a sequence classification head on top (linear layer). [`MistralForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZfddZddZddZeed ej e ej e ej e e ej e ej e ej e ee ee ee eeeefd d d ZZS) MistralForSequenceClassificationcs@t||j|_t||_tj|j|jdd|_| dSr~) rHrI num_labelsrrrrrOscorerrrQr-r.rIws   z)MistralForSequenceClassification.__init__cCs|jjSrr rr-r-r.rsz5MistralForSequenceClassification.get_input_embeddingscCs ||j_dSrr rr-r-r.rsz5MistralForSequenceClassification.set_input_embeddingsNrc  Cs(| dk r | n|jj} |j|||||||| | d } | d} || } |dk rV|jd}n |jd}|jjdkr||dkr|td|jjdkrd}n4|dk rt||jj dd | j }nd}| tj || j d|f}d}|dk r| | j }|jjdkrR|jdkrd|j_n:|jdkrJ|jtj ks@|jtjkrJd |j_nd |j_|jjdkrt}|jdkr|||}n |||}nN|jjd krt}||d|j|d}n|jjd krt}|||}| s|f| dd}|dk r |f|S|St||| j| j| jd S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rr{rrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r)r2Z regressionZsingle_label_classificationZmulti_label_classificationr)rrrr&rtrrr#eqrZargmaxr9r2riZ problem_typer%r rr rwrrrrrrWr)rNrrr{rrrrrrrZtransformer_outputsrWrrZsequence_lengthsZ pooled_logitsrrrr-r-r.rYsr         (    z(MistralForSequenceClassification.forward) NNNNNNNNNN)r[r\r]rIrrrrr#rrrrrrCrrrrYr^r-r-rQr.r$fs6   r$)rr0)N)Irinspectrtypingrrrrr#Ztorch.nn.functionalrrr(Ztorch.utils.checkpointZtorch.nnrrr Z activationsr Zmodeling_outputsr r rZmodeling_utilsrrrrrrrZconfiguration_mistralrZ flash_attnrrZflash_attn.bert_paddingrrrlist signature parametersrZ get_loggerr[rr"r/Sizer r2rr@rrEModulerFr_rvr|r}rrrrZMISTRAL_START_DOCSTRINGrrrr r$r-r-r-r.sz      %  nF@N