U 9%e#@sdZddlZddlmZmZmZddlZddlZddlmZddl m Z m Z m Z m Z ddl mZddlmZmZmZmZmZdd lmZdd lmZmZmZmZmZd d lmZerdd l m!Z!m"Z"ddl#m$Z$m%Z%m&Z&e'e(Z)ddddddgZ*dZ+dZ,Gdddej-Z.ddZ/ddZ0Gdddej1Z2Gdd d e2Z3Gd!d"d"e2Z4ej5ej6e7ej8d#d$d%Z9ej:e7ej8d&d'd(Z;ej:e7eje?ej:d,d-d.Z@Gd/d0d0ej1ZAGd1d2d2eAZBGd3d4d4ej1ZCGd5d6d6ej1ZDd7ZEd8ZFGd9d:d:eZGed;eEGdeEGd?d@d@eGZIedAeEGdBdCdCeGZJedDeEGdEdFdFeGZKedGeEGdHdIdIeGZLdS)JzPyTorch Falcon model.N)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardis_flash_attn_availablelogging) FalconConfig)flash_attn_funcflash_attn_varlen_func)index_first_axis pad_input unpad_inputztiiuae/falcon-40bztiiuae/falcon-40b-instructztiiuae/falcon-7bztiiuae/falcon-7b-instructztiiuae/falcon-rw-7bztiiuae/falcon-rw-1bzRocketknight1/falcon-rw-1brc@s eZdZejejdddZdS) FalconLinear)inputreturncCs$||jj}|jdkr|S||jSN)weightTbias)selfr hidden_statesr'i/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/falcon/modeling_falcon.pyforwardAs  zFalconLinear.forwardN)__name__ __module__ __qualname__torchTensorr)r'r'r'r(r@srcCsJ|dd|jddf|d|jdddf}}tj| |fddS)N.dim)shaper-cat)xx1Zx2r'r'r( rotate_halfIs6r7cCsX|jdtjd}tj|dd}|}ttj |dtjjdd}|||fS)Nr/r2dtypeF)as_tupler)rr) sumr-int32ZnonzeroflattenmaxitemFpadcumsum) padding_maskZseqlens_in_batchindicesZmax_seqlen_in_batchZ cu_seqlensr'r'r(_get_unpad_dataOs rEcsXeZdZdZdedfdd ZddZd ejfeeej ej d d d Z d dZ Z S)FalconRotaryEmbeddingzImplementation of RotaryEmbedding from GPT-NeoX. This implementation is designed to operate on queries and keys that are compatible with `[batch_size, n_heads_per_partition, seq_len, head_dim]` (e.g. MinGPTAttention format). 'head_dimcsbt||_||_d|jtd|d|}|jd|dd||_d|_ d|_ d|_ dS)N?rr0inv_freqF persistentr/) super__init__basemax_position_embeddingsr-arangefloatregister_bufferrJseq_len_cached cos_cached sin_cached)r%rJrQrRrL __class__r'r(rPbs  zFalconRotaryEmbedding.__init__cCs||_tj|||jjd}td||j}tj||fdd|}|tjtj fkrZ| }| dddddf|_ | dddddf|_|j ||_ |j||_dSNdevicer9i,j->ijr/r1)rVr-rSrLr9einsumr4tofloat16bfloat16rTcosrWsinrXtyper%seq_lenr]r9tfreqsembr'r'r(_set_cos_sin_cachemsz(FalconRotaryEmbedding._set_cos_sin_cachecpu)rgpast_key_values_length position_idsr c Csd||}||jkr |||||j||_|j||_|jd|}|jd|}||fS)Nr)rVrkrWr`rXsqueeze) r%rgrmrnr]r9 total_lengthrcrdr'r'r(cos_sin|s zFalconRotaryEmbedding.cos_sincCs|j\}}}|||||j|j\}}t|jd|jd} | dkrhtj|| dd} tj|| dd} n ||} } t|jd|jd} | dkr| | krtj|| dd} tj|| dd}q| | } }n ||} }|| t|| || t||fS)Nrrr1)r3rqr]r9intr-Zrepeat_interleaver7)r%querykeyrmrn_rgrcrdZquery_expansion_factorZ query_cosZ query_sinZkey_expansion_factorZkey_cosZkey_sinr'r'r(r)s    zFalconRotaryEmbedding.forward)rGrH) r*r+r,__doc__rrrPrkr-rbr.rqr) __classcell__r'r'rYr(rF\s  rFcs0eZdZdZd edfdd Zdd ZZS) "FalconLinearScalingRotaryEmbeddingz\FalconRotaryEmbedding extended with linear scaling. Credits to the Reddit user /u/kaiokendevrGrHrKrIcs||_t|||dSr!scaling_factorrOrPr%rJrQrRrzrYr'r(rPsz+FalconLinearScalingRotaryEmbedding.__init__cCs||_tj|||jjd}||j}td||j}tj||fdd|}|tj tj fkrd| }| dddddf|_ |dddddf|_|j ||_ |j||_dSr[)rVr-rSrLr9rzr_r4r`rarbrTrcrWrdrXrerfr'r'r(rks z5FalconLinearScalingRotaryEmbedding._set_cos_sin_cache)rGrHrKr*r+r,rvrrrPrkrwr'r'rYr(rxsrxcs0eZdZdZd edfdd Zdd ZZS) &FalconDynamicNTKScalingRotaryEmbeddingzx FalconRotaryEmbedding extended with Dynamic NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla rGrHrKrIcs||_t|||dSr!ryr{rYr'r(rPsz/FalconDynamicNTKScalingRotaryEmbedding.__init__c Cs$||_||jkrx|j|j||j|jd|j|jd}d|td|jd||j}|j d|ddtj|||j j d}t d ||j }tj ||fd d |}|tjtjfkr|}|dddddf|_|dddddf|_|j||_|j||_dS) Nrr0rKrrLFrMr\r^r/r1)rVrRrQrzrJr-rSrTr`rUrLr9r_r4rarbrcrWrdrXre) r%rgr]r9rQrLrhrirjr'r'r(rks" (z9FalconDynamicNTKScalingRotaryEmbedding._set_cos_sin_cache)rGrHrKr|r'r'rYr(r}sr})input_ids_shaper]rmr cCs||\}}tjtj||ftj|ddd}tj||ftj|d}tj||gdd}|ddddddf|d|||}|S)a Make causal mask used for self-attention. This mask does not take the existing attention mask into account - it just blocks tokens from attending forwards in the sequence. The output shape will be `[batch_size, 1, target_length, target_length+past_key_values_length]`. r9r]r)Zdiagonalr/r1N)r-ZtriuonesboolZzerosr4expand)r~r]rm batch_sizeZ target_lengthmaskZ past_mask expanded_maskr'r'r(_make_causal_masks  (r)rrmr cCsP|j\}}|dk r||n|}|ddddddftj}||d||S)z| Expands attention_mask from `[batch_size, seq_length]` to `[batch_size, 1, seq_length, seq_length + past_length]`. Nr)r3r`r-rr)rrmrrp seq_lengthrr'r'r( _expand_masks "r)attention_mask num_headsr9r cCs:|j\}}dtt|}tjddt|d  |jtjd}tjdd||jtj d}t ||}||krtjddtd|d  |jtjd} t |||} tjddd| d|jtj d} tj |t | | gdd}|j ddd|dddddf} |d| } | ||d||S) Nr0r r\rrr1r/).N)r3mathfloorlog2r-Ztensorr]float32rSr<powminr4rBrbreshaper`)rrr9rrZclosest_power_of_2rQZpowersZslopesZ extra_baseZnum_remaining_headsZ extra_powersZ arange_tensoralibir'r'r(build_alibi_tensors*   &r)r5residualprobtrainingr cCstj|||d}||}|S)a: Dropout add function Args: x (`torch.tensor`, *required*): input tensor residual (`torch.tensor`, *required*): residual tensor prob (`float`, *required*): dropout probability training (`bool`, *required*): training mode )pr)r@dropout)r5rrroutr'r'r( dropout_addsrc seZdZedfdd ZddZejeejejejfdddZ ejejd d d Z deje ejeje ej e eejejfe eje e e ej d ddZZS)FalconAttentionconfigcsDt||_|j|_|j|_|j|j|_|j|_|j|_|j|j|jkrjt d|jd|jd|j rx| ndd|_ dt |j|_|j|_|jr|jd|j|j}n"|jr|jd|j}n d|j}t|j||jd |_|j|_|j|_t|j|j|jd |_t|j|_|js4|js:|jnd |_dS) NzA`hidden_size` must be divisible by num_heads (got `hidden_size`: z and `num_heads`: z).cSs||fSr!r')qkrhrr'r'r(Az*FalconAttention.__init__..rKr0r r$r)rOrPr hidden_sizenum_attention_headsrrJZ split_sizehidden_dropout ValueErrorZrotary _init_rope maybe_rotaryrsqrtinv_norm_factorbetanew_decoder_architecture num_kv_heads multi_queryrr$query_key_valuedenserDropoutattention_dropout)r%rZ qkv_out_dimrYr'r(rP1s2  zFalconAttention.__init__cCs|jjdkr&t|j|jj|jjd}nn|jjd}|jjd}|dkrbt|j|jj|jj|d}n2|dkrt|j|jj|jj|d}ntd||S)N)rQrRrefactorZlinear)rQrRrzZdynamiczUnknown RoPE scaling type ) rZ rope_scalingrFrJZ rope_thetarRrxr}r)r%Z rotary_embZ scaling_typerzr'r'r(rSs0   zFalconAttention._init_rope) fused_qkvr c Cs|jr|j\}}}|||d|j|jd|j}|ddddddddf}|dddddddgf}|dddddddgf}t||j}t||j}dd|||fD\}}}|||fS|js,|j\} } } || | |jd|j}|dd ddf|dd ddf|ddddffS|j\} } } || | |jd|j}|dddddf|ddgddf|ddgddffSdS) a Split the last dimension into (num_heads, head_dim), results share same memory storage as `fused_qkv` Args: fused_qkv (`torch.tensor`, *required*): [batch_size, seq_length, num_heads * 3 * head_dim] Returns: query: [batch_size, seq_length, num_heads, head_dim] key: [batch_size, seq_length, num_heads, head_dim] value: [batch_size, seq_length, num_heads, head_dim] r/r0NcSsg|]}|ddqS)r0r )r=).0r5r'r'r( sz0FalconAttention._split_heads..r .rr) rr3viewrrrJr-Z broadcast_tor) r%rbatchrgruZqkvrsrtvaluerrZthree_times_hidden_sizer'r'r( _split_headsos"      4 zFalconAttention._split_headsr5r cCsP|j\}}}||j}|||j||j}|dddd}||||j|jS)a Merge heads together over the last dimension Args: x (`torch.tensor`, *required*): [batch_size * num_heads, seq_length, head_dim] Returns: torch.tensor: [batch_size, seq_length, num_heads * head_dim] rr0rr )r3rrrJpermuter)r%r5Zbatch_size_and_num_headsrrurr'r'r( _merge_headss  zFalconAttention._merge_headsNF r&rrrn layer_past head_mask use_cacheoutput_attentionsrCc %Cs>||} |jr|jn|j} || \} } }| j\}}}}| dd||j||j} | dd|| ||j} |dd|| ||j}|dkrdn |djd}| | | ||\} } |dk r|\}}t j || fdd} t j ||fdd}| j\}}}|r| |f}nd}t | j j}|d||| j }| ||jd|j}| || d|j}||| d|j}|dkrNttdr|stdtj||||d d d }d}n>||dd }|t|j}tj||d|j d }||}|||j||j}|dddd}||||j|j}||}|rD|||fS||fSn||dd }|||j||}|j } | t jks| t jkr|t j}||||jdd}!|!|j9}!tj|!|d|j d }"| |"}"|dk r|"|}"|"||j||}#|#|!dd}$|"|$}$||$}|r2|||"fS||fSdS)Nrr0rr1rKr/scaled_dot_product_attentionasThe current implementation of Falcon calls `torch.scaled_dot_product_attention` directly, this will be deprecated in the future in favor of the `BetterTransformer` API. Please install the latest optimum library with `pip install -U optimum` and call `model.to_bettertransformer()` to benefit from `torch.scaled_dot_product_attention` and future performance optimizations.F)Z is_causalrr8r )#rrrrrr3 transposerrJrr-r4Zfinfor9rZ masked_fillr`hasattrr@logger warning_oncerrrZsoftmaxrrrrarbrrrr=r)%r%r&rrrnrrrrrCrr query_layer key_layer value_layerr query_lengthrupast_kv_lengthpast_key past_value kv_lengthZpresentZ float_minZattention_mask_floatZ query_layer_Z key_layer_Z value_layer_ attn_outputZattention_scoresZ output_tensorZ matmul_result input_dtypeZattention_logitsZattention_probsZattention_probs_reshapedZ context_layerr'r'r(r)s               zFalconAttention.forward)NNNFFN)r*r+r,rrPrr-r.rrrr LongTensorrr)rwr'r'rYr(r0s*"" rc @sneZdZdZd ejeejejeejeeejejfeeje e eejd ddZ d dd Z d d Z dS)FalconFlashAttention2aH Falcon flash attention module. This module inherits from `FalconAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. NFrc Cs*||} |jr|jn|j} || \} } }| j\}}}}| dd||j||j} | dd|| ||j} |dd|| ||j}|dkrdn |djd}| | | ||\} } |dk r|r|\}}t j || fdd} t j ||fdd}| j\}}}| j }|r| |fnd}| ||jd|jdd |} | || d|jdd |} ||| d|jdd |}|dk rtd|jr|jnd}| j }|t jkrtd| t j} | t j} | t j}|j| | || ||d }||||j|j}||}|s d}|||fS) Nrr0rr1r/z6`alibi` is not supported when `use_flash_attn` is TruerzThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in float16.)r)rrrrrr3rrrJrr-r4r9r`rrrrrrra_flash_attention_forwardr)r%r&rrrnrrrrrCrrrrrrrrurrrZ kv_seq_lengthZ torch_dtypepast_key_valueZ attn_dropoutrrZ attn_weightsr'r'r(r),s`             zFalconFlashAttention2.forwardrc Cs|dk rl|jd}||||||\}}}} } } | \} } | \}}t|||| | ||||dd }t|| ||}nt|||||dd}|S)a Calls the forward method of Flash Attention - if the input hidden states contain at least one padding token first unpad the input, then computes the attention scores and pad the final attention scores. Args: query_states (`torch.Tensor`): Input query states to be passed to Flash Attention API key_states (`torch.Tensor`): Input key states to be passed to Flash Attention API value_states (`torch.Tensor`): Input value states to be passed to Flash Attention API padding_mask (`torch.Tensor`): The padding mask - corresponds to a tensor of size `(batch_size, seq_len)` where 0 stands for the position of padding tokens and 1 for the position of non-padding tokens. dropout (`int`, *optional*): Attention dropout softmax_scale (`float`, *optional*): The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim) NrT) cu_seqlens_q cu_seqlens_kZ max_seqlen_qZ max_seqlen_kZ dropout_p softmax_scalecausal)rr)r3 _upad_inputrrr)r%Z query_statesZ key_statesZ value_statesrCrrrr indices_qZ cu_seq_lensZ max_seq_lensrrmax_seqlen_in_batch_qmax_seqlen_in_batch_kZattn_output_unpadrr'r'r(rsB  z.FalconFlashAttention2._flash_attention_forwardcCst|\}}}|j\} } } } t|| | | | |}t|| | | | |}|| kr|t|| | |j| |}|} |}|}nd|dkrd}tj| dtj|jd} | dd}| d}n(|dd| df}t ||\}}} }||||| |f||ffS)Nrrr/) rEr3rrrr-rSr<r]ror)r%rrrrCrZ indices_krrrZ kv_seq_lenZnum_key_value_headsrJrrrr'r'r(rsH  z!FalconFlashAttention2._upad_input)NNNFFN)rN) r*r+r,rvr-r.rrrrr)rrr'r'r'r(r%s,  T 6rcs6eZdZedfdd ZejejdddZZS) FalconMLPrcsRt|j}t|d||jd|_t|_td|||jd|_ |j |_ dS)Nr) rOrPrrr$ dense_h_to_4hrZGELUact dense_4h_to_hrr%rrrYr'r(rPs   zFalconMLP.__init__rcCs|||}||}|Sr!)rrr)r%r5r'r'r(r)s zFalconMLP.forward) r*r+r,rrPr-r.r)rwr'r'rYr(rs rc sneZdZedfdd Zd ejeejejeejee ejejfeeje e eejd ddZ Z S) FalconDecoderLayerrcst|j}|j|_t|dds,t|nt||_t ||_ |j |_ ||_ |j rvt||jd|_t||jd|_n&t||jd|_|jst||jd|_dS)NZ_flash_attn_2_enabledFZeps)rOrPrrrgetattrrrself_attentionrmlprrrrlayer_norm_epsilonln_attnln_mlpinput_layernorm parallel_attnpost_attention_layernormrrYr'r(rPs    zFalconDecoderLayer.__init__NFrc  Cs|} |jjr"||} ||} n ||} |j| |||||||| d } | d}|jjs|jjrf| } n t|| |jj|j d} | | } | dd}| | }|jjs|jjr||7}t|| |jj |j d}|r|f|}n|f|dd}|S)N)rrrnrrrrrCr)rr) rrrrrrrrrrrrr)r%r&rrrnrrrrrCrZattention_layernorm_outZmlp_layernorm_outZ attn_outputsZattention_outputoutputsZ mlp_outputoutputr'r'r(r)sH         zFalconDecoderLayer.forward)NNNFFN) r*r+r,rrPr-r.rrrrr)rwr'r'rYr(rs$ra- This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`FalconConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a2 Args: input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0][0].shape[2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) past_key_values (`Tuple[Tuple[torch.Tensor]]` of length `config.num_hidden_layers`): Contains precomputed hidden-states (key and values in the attention blocks) as computed by the model (see `past_key_values` output below). Can be used to speed up sequential decoding. The `input_ids` which have their past given to this model should not be passed as `input_ids` as they have already been computed. Each element of `past_key_values` is a tuple (past_key, past_value): - past_key: [batch_size * num_heads, head_dim, kv_length] - past_value: [batch_size * num_heads, kv_length, head_dim] attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `past_key_values` is used, optionally only the last `inputs_embeds` have to be input (see `past_key_values`). use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. cseZdZdZeZdZdZdgZdZ fddZ e j ddd Z de j ed d d ZeeeejejfeeeejejfdddZeeeejejfeeejejfdddZZS)FalconPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. transformerTrcstj||dSr!)rOrP)r%inputskwargsrYr'r(rPszFalconPreTrainedModel.__init__modulecCst|tjst|trD|jjjd|jjd|j dk r|j j ndt|tj r|jjjd|jjd|j dk r|jj|j  n$t|t r|j j |jjddS)zInitialize the weights.r)ZmeanZstdNrK) isinstancerLinearrr"dataZnormal_rZinitializer_ranger$Zzero_ EmbeddingZ padding_idxrZfill_)r%rr'r'r( _init_weightss     z#FalconPreTrainedModel._init_weightsF)rrcCst|tr||_dSr!)r FalconModelgradient_checkpointing)r%rrr'r'r(_set_gradient_checkpointings z1FalconPreTrainedModel._set_gradient_checkpointing)rrr cs8|ddj\}|tfdd|DS)z Standardizes the format of the cache so as to match most implementations, i.e. to tuple(tuple([batch_size, num_heads, ...])) rc3s6|].}|d|dfVqdSrrNrrrrrJrrr'r( szJFalconPreTrainedModel._convert_cache_to_standard_format..r3tuple)rrbatch_size_times_num_headsr'rr(!_convert_cache_to_standard_formats z7FalconPreTrainedModel._convert_cache_to_standard_format)rr cs8|ddj\}}||tfdd|DS)Nrc3s2|]*}|d|dfVqdSrrrrrJrr'r(rsz=FalconPreTrainedModel._convert_to_rw_cache..r)rrrr'r r(_convert_to_rw_caches z*FalconPreTrainedModel._convert_to_rw_cache)F)r*r+r,rvr config_classZbase_model_prefixZsupports_gradient_checkpointingZ_no_split_modulesZ_supports_flash_attn_2rPrModulerrr staticmethodrr-r.rrr r rwr'r'rYr(rs$ rz`The bare Falcon Model transformer outputting raw hidden-states without any specific head on top.cseZdZedfdd ZddZeeje e e fe ej dddZ ejd d d Z eeeeeed deejee e ejejfdfeejeejeejeejeeeeeeeeee ejdfefd ddZZS)rrcsztj|_j|_j|_t j |j|_ t fddt jD|_t|jjd|_d|_|dS)Ncsg|] }tqSr')r)rrurr'r(rsz(FalconModel.__init__..rF)rOrPrZ embed_dimrrr use_alibirr vocab_sizeword_embeddingsZ ModuleListrangenum_hidden_layershrrln_fr post_initr%rrYrr(rPs  zFalconModel.__init__cCs|jSr!rr%r'r'r(get_input_embeddingssz FalconModel.get_input_embeddings)r input_shapermr cCs|d||jdkr4td|jd|d|dd}|j}|\}}|dkr\t|||d}t||d}|dkrt|n||B}|S)NrzXAttention mask shape should be (batch_size, seq_length + past_key_values_length) but is z with input_ids shape z and past length .)r]rm)rm)r3rr]rr)rrrmZcombined_attention_maskr]rurZexpanded_attn_maskr'r'r(_prepare_attn_masks" zFalconModel._prepare_attn_masknew_embeddingscCs ||_dSr!rr%rr'r'r(set_input_embeddingssz FalconModel.set_input_embeddings checkpoint output_typer N.) input_idspast_key_valuesrrnr inputs_embedsrroutput_hidden_states return_dictr c  sNdk r n|jj| dk r | n|jj} dk r4n|jj| dk rH| n|jj} |dk rj|dk rjtdn2|dk r~|j\} } n|dk r|j\} } } ntd|dkrtdgt|j }n | |}| ||jj }|dkr| |}|}rdnd}rdnd}| r dnd}d}|ddk r4|ddjd}|dkr\tj| | |f|jd}d}n ||j}d|krx|}nd}|jrt||j|jd}n`d}|dkr|dk r|jn|j}tj|| |tj|d}|dd | }n|d | }|j|| | f|d }tt|j |D]\}\}}| r8||f}|jr|jrr\t d d fd d}tj!j""|||||||||}n|||||||||d }|d}dkr||df}r||rdndf}q|#|}| r||f}|dk r|$|| }| s>tdd||||fDSt%||||dS)NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr'rrr])r9rr/)rrmzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fcsfdd}|S)Ncs|dS)N)rrr')r)rrrr'r(custom_forward{szJFalconModel.forward..create_custom_forward..custom_forwardr')rr+rrrr(create_custom_forwardzsz2FalconModel.forward..create_custom_forward)rrrnrrrrrCTr0css|]}|dk r|VqdSr!r')rvr'r'r(rsz&FalconModel.forward..)Zlast_hidden_stater&r& attentions)&rrr(ruse_return_dictrr3rlenrr Z get_head_maskrrr-rr]r`rrrr9rSlong unsqueezerr enumerateziprrrwarningutilsr#rr r )r%r%r&rrnrr'rrr(r)rrrur&ZpresentsZall_self_attentionsZall_hidden_statesrmrCrr]Z causal_maskiblockrr-rr'r,r(r)s                  zFalconModel.forward) NNNNNNNNNN)r*r+r,rrPrrr-r.rrr BoolTensorrr!rFALCON_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrrrrr)rwr'r'rYr(rsL rz{The Falcon Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings).cs>eZdZdgZedfdd ZddZejddd Z dej e eje eje eje d d d Z eeeeeedde ej e eeejejfdfe eje ej e eje eje eje ee ee ee eeeejefd ddZeeejejfdfej eeejejfdfdddZZS)FalconForCausalLMzlm_head.weightrcs8t|t||_tj|j|jdd|_| dSNFr) rOrPrrrrrrlm_headrrrYr'r(rPs  zFalconForCausalLM.__init__cCs|jSr!r@rr'r'r(get_output_embeddingssz'FalconForCausalLM.get_output_embeddingsrcCs ||_dSr!rAr r'r'r(set_output_embeddingssz'FalconForCausalLM.set_output_embeddingsN)r%r&rrnr cKs|dk r|ddddf}|jjsp|dk rp|dkrp|dd}||dkd|rp|dddfd}||||d|dS)Nr/rrr)r%rnr&rr)rrr2rBZ masked_fill_r3get)r%r%r&rrnrr'r'r(prepare_inputs_for_generationsz/FalconForCausalLM.prepare_inputs_for_generationr".) r%r&rrnrr'labelsrrr(r)r c  Cs| dk r | n|jj} |j|||||||| | | d } | d} || }d}|dk r|dddddf}|dddf}|j\}}}t}||||||||}| s|f| dd}|dk r|f|S|St||| j | j | j dS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N) r&rrnrr'rrr(r)r.r/rlosslogitsr&r&r/) rr0rr@ contiguousr3rrr r&r&r/)r%r%r&rrnrr'rFrrr(r)transformer_outputsr&Z lm_logitsrHZ shift_logitsZ shift_labelsrrrloss_fctrr'r'r(r)sF    zFalconForCausalLM.forward)pastbeam_idxr cs,fdd|Dtfdd|D}|S)aL This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct beam_idx at every generation step. Output shares the same memory storage as `past`. cs&i|]}|D]}|j|jq qSr')r]r`)rrZ past_state)rNr'r( 3s z4FalconForCausalLM._reorder_cache..c3sB|]:}|dd|dj|dd|djfVqdSr)Z index_selectr]r)device_to_beam_idxr'r(r6sz3FalconForCausalLM._reorder_cache..)r)r%rMrNZreordered_pastr')rNrPr(_reorder_cache's  z FalconForCausalLM._reorder_cache)NNN) NNNNNNNNNNN)r*r+r,Z_tied_weights_keysrrPrBr-r.rCrrdictrErr;rr<r r=rrrr)rQrwr'r'rYr(r>sb @r>a The Falcon Model transformer with a sequence classification head on top (linear layer). [`FalconForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). cseZdZedfdd Zeeeee e dd e e j e eee je jfdfe e je e je e je e je ee ee ee eeee je fd dd ZZS) FalconForSequenceClassificationrcs@t||j|_t||_tj|j|jdd|_| dSr?) rOrP num_labelsrrrrrscorerrrYr'r(rPPs   z(FalconForSequenceClassification.__init__r"N. r%r&rrr'rFrrr(r)r c  Cs| dk r | n|jj} |j|||||||| | d } | d} || } |dk rV|jd}n |jd}|jjdkr||dkr|td|jjdkrd}nF|dk rt||jjj ddd | j }nd}t |jjd| tj|| j d |f}d}|dk r|jjdkrX|jdkrd |j_n:|jdkrP|jtjksF|jtjkrPd |j_nd |j_|jjd krt}|jdkr|||}n |||}n>|jjd krt}|||}n|jjd krt}|||}| s|f| dd}|dk r|f|S|St||| j| j| jd S) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr&rrr'rrr(r)rrz=Cannot handle batch sizes > 1 if no padding token is defined.r/r1z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r*Z regressionZsingle_label_classificationZmulti_label_classificationrG)rr0rrUr3Z pad_token_idrr-ner;r`r]rr6rZr*rSZ problem_typerTr9r2rrr rorrrr&r&r/)r%r%r&rrr'rFrrr(r)rKr&rIrZsequence_lengthsZ pooled_logitsrHrLrr'r'r(r)Ysr     &    (     z'FalconForSequenceClassification.forward) NNNNNNNNNN)r*r+r,rrPrr;rr<rr=rr-rrr.rrr)rwr'r'rYr(rS@s< rSz Falcon Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. cseZdZedfdd Zeeeee e dd e e j e eee je jfdfe e je e je e je e je ee ee ee eeee je fd dd ZZS) FalconForTokenClassificationrcs|t||j|_t||_t|dddk r6|j}nt|dddk rN|j}nd}t ||_ t |j |j|_ |dS)Nclassifier_dropoutrg?)rOrPrTrrrr[rrrrrr classifierr)r%rr[rYr'r(rPs   z%FalconForTokenClassification.__init__r"N.rVc  Cs| dk r | n|jj} |j|||||||| | d } | d} || } || } d}|dk r|j\}}t}|| |||j|||}| s| f| dd}|dk r|f|S|St || | j | j dS)rWNrXrr0)rHrIr&r/) rr0rrr\r3rrrTrr&r/)r%r%r&rrr'rFrrr(r)rKr&rIrHrrrLrr'r'r(r)s@     z$FalconForTokenClassification.forward) NNNNNNNNNN)r*r+r,rrPrr;rr<rr=rr-rrr.rrr)rwr'r'rYr(rZs<rZz The Falcon Model transformer with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layers on top of the hidden-states output to compute `span start logits` and `span end logits`). cs~eZdZfddZeedeejeej eej eej eejeejee ee ee e e e fd ddZZS)FalconForQuestionAnsweringcs2t|t||_t|jd|_|dS)Nr0) rOrPrrrrr qa_outputsrrrYr'r(rPs  z#FalconForQuestionAnswering.__init__N) r%rrr'start_positions end_positionsrr(r)r c  CsL| dk r | n|jj} |j||||||| d} | d} || } | jddd\} }| d} |d}d}|dk r|dk rt|dkr|d}t|dkr|d}| d}| d|}| d|}t |d}|| |}|||}||d}| s6| |f| dd}|dk r2|f|S|St || || j | j d S) a start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. N)rrr'rr(r)rrr/r1)Z ignore_indexr0)rH start_logits end_logitsr&r/)rr0rr^splitrorJr1sizeclamprrr&r/)r%r%rrr'r_r`rr(r)rZsequence_outputrIrarbZ total_lossZ ignored_indexrLZ start_lossZend_lossrr'r'r(r)%sL           z"FalconForQuestionAnswering.forward) NNNNNNNNN)r*r+r,rPrr;rr-rZ FloatTensorrrrrr)rwr'r'rYr(r]s.  r])Mrvrtypingrrrr-Ztorch.utils.checkpointrZtorch.nnrrrr r r@Zmodeling_outputsr r rrrZmodeling_utilsrr7rrrrrZconfiguration_falconrZ flash_attnrrZflash_attn.bert_paddingrrrZ get_loggerr*rZ$FALCON_PRETRAINED_MODEL_ARCHIVE_LISTr<r=rrr7rEr rFrxr}Sizer]rrr:rr.rr9rrTrrrrrrZFALCON_START_DOCSTRINGr;rrr>rSrZr]r'r'r'r(s       I#  v8W<IX  mR