U 9%e>@sdZddlZddlmZmZmZmZmZmZddl Z ddl m Z m Z ddl m Z mZddlmZddlmZdd lmZmZmZmZdd lmZdd lmZmZmZmZmZm Z d d l!m"Z"e#e$Z%dZ&dZ'dZ(dZ)dZ*ddZ+d;ddZ,dde j-fddZ.GdddeZ/ddZ0ddZ1dd Z2dGd9d:d:e j?Z@dS)=z`PyTorch Fairseq model, ported from https://github.com/pytorch/fairseq/tree/master/examples/wmt19N)AnyDictListOptionalTupleUnion)Tensornn)CrossEntropyLoss LayerNorm)ACT2FN)is_deepspeed_zero3_enabled)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)add_code_sample_docstringsadd_end_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) FSMTConfigzfacebook/wmt19-ru-enra? This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`FSMTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. u Translation example:: ```python >>> from transformers import AutoTokenizer, FSMTForConditionalGeneration >>> mname = "facebook/wmt19-ru-en" >>> model = FSMTForConditionalGeneration.from_pretrained(mname) >>> tokenizer = AutoTokenizer.from_pretrained(mname) >>> src_text = "Машинное обучение - это здорово, не так ли?" >>> input_ids = tokenizer(src_text, return_tensors="pt").input_ids >>> outputs = model.generate(input_ids, num_beams=5, num_return_sequences=3) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) "Machine learning is great, isn't it?" ``` a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`FSTMTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) FSMT uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. encoder_outputs (`Tuple(torch.FloatTensor)`, *optional*): Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`) `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)` is a sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. past_key_values (`Tuple(torch.FloatTensor)` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden-states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be input (see `past_key_values`). This is useful if you want more control over how to convert `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix. If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value of `inputs_embeds`. use_cache (`bool`, *optional*, defaults to `True`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cCs|dkst|dS)z+Turns 1->0, 0->1, False->True, True-> Falser)dimAssertionErroreq)attention_maskr!e/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/fsmt/modeling_fsmt.py invert_mask+sr#cCsT|jd}tj||jd}|||}|d}|r<||}||k}||dkdS)Nrdevice)shapetorcharanger%expand unsqueeze masked_fill)xZdiagonallr)maskr!r!r" triu_onnx1s   r0c Csn|j}|dkrt||}|\}}|dkr8t||}nt|}tttj|||ddj |j d}|||fS)z Prepare masks that ignore padding tokens in the decoder and a causal mask for the decoder if none are provided. This mimics the default behavior in fairseq. To override it pass in masks. Note: this is not called during generation Ndtyperr$) pad_token_idshift_tokens_rightsizemake_padding_maskr#r0fill_with_neg_infr(zerostor%) config input_idsdecoder_input_idsdecoder_padding_maskcausal_mask_dtyper3bsztgt_len causal_maskr!r!r"_prepare_fsmt_decoder_inputs<s    rBc@s(eZdZeZdZddZeddZdS)PretrainedFSMTModelmodelcCs|jj}t|tjr>|jjjd|d|jdk r|jj nFt|t rJn:t|tj r|jjjd|d|j dk r|jj|j  dS)N)Zmeanstd) r:Zinit_std isinstancer LinearweightdataZnormal_biasZzero_SinusoidalPositionalEmbedding Embedding padding_idx)selfmodulerFr!r!r" _init_weightsZs     z!PretrainedFSMTModel._init_weightscCsD|jj}tjdddddgdddd|gg|jd}|||d }|S) Nr r r$)r r;)r:r3r(tensorr%ne)rOZ pad_tokenr; dummy_inputsr!r!r"rYgs (z PretrainedFSMTModel.dummy_inputsN) __name__ __module__ __qualname__r config_classbase_model_prefixrQpropertyrYr!r!r!r"rCVs  rCcCs,|jj\}}tj||dd}|jj|j_|S)NFrK)rIr'r rHrJ)embZ vocab_sizeZemb_sizeZ lin_layerr!r!r"_make_linear_from_embrs  rbcCs ||krtd|d|dS)Nzshape mismatch: z != )r)Zshape_1Zshape2r!r!r" _check_shapeszsrccCsx||dk||}||jdddd}|d||dddf<|ddddf|ddddf<|S)zXShift input ids one token to the right, and wrap the last non pad token (usually ).irrr&Nr)Z masked_fill_clonerXsumr+gatherZsqueeze)r;r3Zprev_output_tokensZ index_of_eosr!r!r"r4s $r4cCs||}|sd}|S)zTrue for pad tokensN)rany)r;rNZ padding_maskr!r!r"r6s r6cs,eZdZedfdd ZdddZZS) EncoderLayerr:cst|j|_t|j|j|jd|_t|j|_ |j |_ t |j |_ |j|_t|j|j|_t|j|j|_t|j|_dS)N)dropout)super__init__d_model embed_dim AttentionZencoder_attention_headsattention_dropout self_attnr self_attn_layer_normrkr activation_function activation_fnactivation_dropoutr rHZencoder_ffn_dimfc1fc2final_layer_normrOr: __class__r!r"rms   zEncoderLayer.__init__FcCs|}|j|||||d\}}tjj||j|jd}||}||}|}|||}tjj||j|jd}| |}tjj||j|jd}||}| |}||fS)al Args: x (`torch.Tensor`): input to the layer of shape *(seq_len, batch, embed_dim)* encoder_padding_mask (`torch.ByteTensor`): binary ByteTensor of shape *(batch, src_len)* where padding elements are indicated by `1`. for t_tgt, t_src is excluded (or masked out), =0 means it is included in attention layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size *(config.encoder_attention_heads,)*. Returns: encoded output of shape *(seq_len, batch, embed_dim)* )querykeykey_padding_masklayer_head_maskoutput_attentionsptraining) rrr functionalrkrrsrurwrvrxry)rOr-encoder_padding_maskrrresidual attn_weightsr!r!r"forwards&    zEncoderLayer.forward)FrZr[r\rrmr __classcell__r!r!r{r"ris ric sReZdZdZedfdd Zd ejeejejeeje e e dd d Z Z S) FSMTEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`EncoderLayer`]. Args: config: FSMTConfig rjcstj|_j|_|j|_||_|j}jr>t |nd|_ t j |jd||j|_tfddtjD|_dS)N?rcsg|] }tqSr!)ri.0_rjr!r" sz(FSMTEncoder.__init__..)rlrmrkZencoder_layerdrop layerdroprN embed_tokens embedding_dimscale_embeddingmathsqrt embed_scalerLmax_position_embeddingsembed_positionsr ModuleListrangeZencoder_layerslayers)rOr:rror{rjr"rms zFSMTEncoder.__init__NFTr;r inputs_embeds head_maskroutput_hidden_states return_dictcCs|dk rt|}|dk r*|dk r*tdn|dk rN|||j}||}n`|dk r||j}|dddddf|dddddfd|jj} || }ntd||} tj j | |j |j d} | dd} |rdnd} |rdnd} |dk r6| dt|jks6tdt|jd | dd t|jD]\} }|rp| dd} | | f7} | dd} tg}|j r||jkrd}n&|| ||dk r|| nd|d \} }|r@| |f} q@| dd} |r| | f7} |std d | | | fDSt| | | dS)a Args: input_ids (`torch.LongTensor`): tokens in the source language of shape *(batch, src_len)* attention_mask (`torch.LongTensor`): indicating which indices are padding tokens inputs_embeds (`torch.FloatTensor`): embedding vectors of shape *(batch, src_len, embed_dim)* head_mask (`torch.Tensor` of shape `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Returns: BaseModelOutput or Tuple comprised of: - **x** (`torch.Tensor`): the last encoder layer's output of shape *(src_len, batch, embed_dim)* - **encoder_states** (`Tuple(torch.FloatTensor`)): all intermediate hidden states of shape *(src_len, batch, embed_dim)*. Only populated if *output_hidden_states:* is True. - **all_attentions** (`Tuple(torch.FloatTensor`)): Attention weights for each layer. During training might not be of length n_layers because of layer dropout. NzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrr!z&The head_mask should be specified for  layers, but it is for .)rrcss|]}|dk r|VqdSNr!rvr!r!r" =sz&FSMTEncoder.forward..last_hidden_state hidden_states attentions)r# ValueErrorrrrr,rrNr rrkr transposer5lenrr enumerater(randrtupler)rOr;r rrrrrZ embed_pos position_idsr-Zencoder_statesZall_attentionsidxZ encoder_layerdropout_probabilityZattnr!r!r"rs`!                zFSMTEncoder.forward)NNNFFT) rZr[r\__doc__rrmr(rrboolrrr!r!r{r"rs"rcs,eZdZedfdd ZdddZZS) DecoderLayerrjcst|j|_t|j|j|jd|_|j|_t |j |_ |j |_ t |j|_t|j|j|jdd|_t |j|_t|j|j|_t|j|j|_t |j|_dS)N)ro num_headsrkT)rkencoder_decoder_attention)rlrmrnrorpZdecoder_attention_headsrqrrrkr rtrurvr rs encoder_attnencoder_attn_layer_normr rHZdecoder_ffn_dimrwrxryrzr{r!r"rmBs*    zDecoderLayer.__init__NFc Cs|} |dkri}|j||||||| d\}} tjj||j|jd}| |}||}|} |jj|jjkslt|j|||||| d\}} tjj||j|jd}| |}| |}|} | | |}tjj||j |jd}| |}tjj||j|jd}| |}||}|| || fS)N)r}r~ layer_stater attn_maskrrr)r}r~rrrr)rrr rrkrrsr cache_keyrrrurwrvrxry) rOr-encoder_hidden_statesencoder_attn_maskrrArcross_attn_layer_head_maskr=rrZself_attn_weightsZcross_attn_weightsr!r!r"r[sP       zDecoderLayer.forward)NNNNNNFrr!r!r{r"rAsrcsxeZdZdZeejdfdd Zd ej ej ej ej ej e ej e ej e ej e e ej e e e e d d d ZZS) FSMTDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`DecoderLayer`] Args: config: FSMTConfig embed_tokens (nn.Embedding): output embedding )r:rc stj|_j|_|j|_jr4tj nd|_ ||_ |j }t j|jd||j|_tfddtjD|_trddl}|jj|j jdd|j jj}W5QRXn |j jj}tj|d|ddd|_|j j|j_dS) Nrrcsg|] }tqSr!)rrrjr!r"rsz(FSMTDecoder.__init__..r)Z modifier_rankFr`)rlrmrkZdecoder_layerdroprrNrrrrnrrrrLrrr rrZdecoder_layersrr deepspeedzeroZGatheredParametersrIr'rHoutput_projection)rOr:rrorZembed_tokens_weight_shaper{rjr"rms,  zFSMTDecoder.__init__NFT) r;rrr=decoder_causal_maskrrcross_attn_head_maskpast_key_values use_cacherrrc Cs|dk rt|}|dk r*|dk r*tdn|dk rz||}| rh|ddddf}|ddddf}|||j}n`|dk r|dddddf|dddddfd|jj}||}||j}ntd||7}tj j ||j |j d}| dd}| dd}| rdnd}| r(dnd}| r6dnd}g}t ||gd d gD]V\}}|dk rP|dt|jksPtd |d t|jd |ddqPt|jD]\}}| r| dd}||f7}| dd}|j rtg}||jkrq| dk r| |nd}||||||||dk r:||nd|dk rN||nd| d \}}}}| rt||| r||f7}||f7}q| r| dd}||f7}| dd}| dd}| dd}||}| r|nd}| s tdd|||||fDSt|||||dS)a Includes several features from "Jointly Learning to Align and Translate with Transformer Models" (Garg et al., EMNLP 2019). Args: input_ids (`torch.LongTensor` of shape `(batch, tgt_len)`): previous decoder outputs for teacher forcing encoder_hidden_states: output from the encoder, used for encoder-side attention encoder_padding_mask: for ignoring pad tokens past_key_values (dict or None): dictionary used for storing state during generation head_mask (`torch.Tensor` of shape `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Returns: BaseModelOutputWithPast or tuple: - the decoder's features of shape *(batch, tgt_len, embed_dim)* - the cache - hidden states - attentions NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer&rzEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrr!rrzThe `z` should be specified for rr)rr=rrArrrcss|]}|dk r|VqdSrr!rr!r!r"rEsz&FSMTDecoder.forward..)rrrrcross_attentions)r#rrrrr,rrNr rrkrrzipr5rrrrr(rrappendcopyrrr)rOr;rrr=rrrrrrrrr positionsr-rZall_hidden_statesZall_self_attnsZall_cross_attnsZnext_decoder_cacherZ mask_namerZ decoder_layerrrZlayer_self_attn layer_pastZlayer_cross_attnZ next_cacher!r!r"rs0       $              zFSMTDecoder.forward)NNNNFFFT)rZr[r\rrr rMrmr(rrr FloatTensorrrrr!r!r{r"rs2  rcCs.|D] \}}|dk r|d|||<q|S)Nr)itemsZ index_select) attn_cacheZ new_orderkZinput_buffer_kr!r!r"_reorder_bufferQsrc sxeZdZdZdfdd ZddZdeeeeeee eefeeeee eeefd d d Z d dZ Z S)rpz=Multi-headed attention from 'Attention Is All You Need' paperrETFcst||_||_||_|||_|j||jks>td|jd|_||_t j |||d|_ t j |||d|_ t j |||d|_ t j |||d|_|jrdnd|_dS)Nz(embed_dim must be divisible by num_headsgr`Zencoder_decoderrO)rlrmrorrkhead_dimrscalingrr rHk_projv_projq_projout_projr)rOrorrkrKrr{r!r"rm[s   zAttention.__init__cCs"||||j|jddS)Nrr) contiguousviewrrr)rOrWseq_lenr?r!r!r"_shaperszAttention._shapeN)r~rrrrreturncCs|j}|\} } } | |jks"tt|| | | gks)NNzIMake sure that `decoder_input_ids` or `decoder_inputs_embeds` are passed.rrrrr) rrrrrrrrr)rrdecoder_hidden_statesdecoder_attentionsrencoder_last_hidden_staterencoder_attentions)r:rrruse_return_dictrBrrrIr2rrrGrrrrrrrr)rOr;r r<rrrrrrrrrrrrr=rAZdecoder_outputsr!r!r"r'sz   zFSMTModel.forwardcCs|jjSrrrrr!r!r"rszFSMTModel.get_input_embeddingscCs ||j_dSrr rOvaluer!r!r"set_input_embeddingsszFSMTModel.set_input_embeddingscCs|jjSrrrrr!r!r"get_output_embeddingsszFSMTModel.get_output_embeddingscCs ||j_dSrrr r!r!r"set_output_embeddingsszFSMTModel.set_output_embeddings)NNNNNNNNNNNNNN)rZr[r\_tied_weights_keysrrmrrrrFSMT_INPUTS_DOCSTRINGr_CHECKPOINT_FOR_DOCr_CONFIG_FOR_DOCr( LongTensorrr BoolTensorrrrrrrrrrrr!r!r{r"rs^   \rzLThe FSMT Model with a language modeling head. Can be used for summarization.cs*eZdZdZddgZedfdd Zeee e e de e dejeejeejeejeejeejeejeeejeeejeejeejeejeeeeeeeeeeeje fd d d Zdd d ZejdddZeddZddZddZddZddZZ S)FSMTForConditionalGenerationrDrrrjcs&t|t|}||_|dSr)rlrmrrDr)rOr:Z base_modelr{r!r"rms z%FSMTForConditionalGeneration.__init__)rr]N)r;r r<rrrrrrrrlabelsrrrrrcCs|dk r |n|jj}| dk r d} |j|| ||| |||||| | |||d}|d}d}| dk r~t}||d|jj| d}|s|f|dd}|dk r|f|S|St|||j|j|j |j |j |j |j d S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Returns: NF)rr r<rrrrrrrrrrrrr&r) ZlossZlogitsrrrrrrr )r:r rDr rrrrrrrrrr )rOr;r r<rrrrrrrrrrrrroutputsZ lm_logitsZmasked_lm_lossZloss_fctoutputr!r!r"rsNz$FSMTForConditionalGeneration.forwardc Ksd||||||||d S)N) r;rrr<r rrrrr!) rOr<rr rrrrrkwargsr!r!r"prepare_inputs_for_generations z:FSMTForConditionalGeneration.prepare_inputs_for_generation)rcCst||jjSr)r4r:r3)rOrr!r!r"%prepare_decoder_input_ids_from_labels szBFSMTForConditionalGeneration.prepare_decoder_input_ids_from_labelscs2g}|D]$}fdd|D}||q|S)Ncsi|]\}}|t|qSr!)r)rZattn_keyrbeam_idxr!r" sz?FSMTForConditionalGeneration._reorder_cache..)rr)rr Zreordered_pastrZlayer_past_newr!rr"_reorder_cache s  z+FSMTForConditionalGeneration._reorder_cachecCs|jjSr)rDrrr!r!r"rsz(FSMTForConditionalGeneration.get_encodercCs|jjSr)rDrrr!r!r"rsz(FSMTForConditionalGeneration.get_decodercCs |jjjSrrDrrrr!r!r"rsz2FSMTForConditionalGeneration.get_output_embeddingscCs||jj_dSrr#r r!r!r"r!sz2FSMTForConditionalGeneration.set_output_embeddings)NNNNNNNNNNNNNNN)NNNNNNN)!rZr[r\r^rrrmrrrrrrFSMT_GENERATION_EXAMPLEr(rrrrrrrrrrr staticmethodr"rrrrrr!r!r{r"rsn   M  rcsdeZdZdZddZfddZeddZeedd d Z de e e e d fd d Z ZS)rLa< This module produces sinusoidal positional embeddings of any length. We don't want to save the weight of this embedding since it's not trained (deterministic) and it can be huge. Padding symbols are ignored. These embeddings get automatically extended in forward if more positions is needed. cCs||||dSr) make_weight)rO num_positionsrrNr!r!r"rm0sz&SinusoidalPositionalEmbedding.__init__csf||||}t|ds.tj||||dn"|j|jj|jjd}t ||_|j d|j_ dS)NrI)Z_weight)r2r%F) get_embeddinghasattrrlrmr9rIr2r%r ParameterZdetach_Z requires_grad)rOr'rrNrIr{r!r"r&3s   z)SinusoidalPositionalEmbedding.make_weightcCs|d}td|d}ttj|tjd| }tj|tjdd|d}tjt|t |gdd |d}|ddkrtj|t |dgdd}|dk rd||ddf<|S) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'rr1rrdr&N) rlogr(expr)rr+rsincosrr8)Znum_embeddingsrrNZhalf_dimrar!r!r"r(?s & z+SinusoidalPositionalEmbedding.get_embedding)rNcCs.||}tj|dd|||S)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. rrd)rXintr(Zcumsumrlong)rWrNr/r!r!r"make_positionsSs z,SinusoidalPositionalEmbedding.make_positionsN)incremental_statetimestepcs\|jdd\}}|jd|}||jdkrB|||j|j|||j}t|S)z/Input is expected to be of size [bsz x seqlen].Nrrr) r'rNrIr5r&rr1rlr)rOinputr2r3r?rZmax_posrr{r!r"ras z%SinusoidalPositionalEmbedding.forward)NN)rZr[r\rrmr&r%r(r/r1rrrrrr!r!r{r"rL%s   rL)r)r)Arrtypingrrrrrrr(rr Ztorch.nnr r Z activationsr Zintegrations.deepspeedrZmodeling_outputsrrrrZmodeling_utilsrutilsrrrrrrZconfiguration_fsmtrZ get_loggerrZloggerrrZFSMT_START_DOCSTRINGr$rr#r0Zfloat32rBrCrbrcr4r6Modulerirrrrrpr7rrrrMrLr!r!r!r"sf       yS   1yX9'