U *-e@sddlZddlmZmZmZmZddlZddlZddlmZddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZdddddgZeejefeejej edddZ!ee"eedddZ#Gddde Z$Gddde Z%Gddde Z&Gddde Z'Gddde Z(ddZ)e*eegefd d!d"Z+d&eeee"eee"d#d$d%Z,dS)'N)OptionalAnyUnionCallable)Tensor) functional)Module)MultiheadAttention) ModuleList)xavier_uniform_)Dropout)Linear) LayerNorm TransformerTransformerEncoderTransformerDecoderTransformerEncoderLayerTransformerDecoderLayerszdevicedtypereturncCs$tjtj||ftd||dddS)zGenerate a square causal mask for the sequence. The masked positions are filled with float('-inf'). Unmasked positions are filled with float(0.0). z-infrrr )Zdiagonal)torchZtriufullfloatrrrr ]/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torch/nn/modules/transformer.py _generate_square_subsequent_masksr")src batch_firstrcCs>|jr dS|}t|dkr&|dS|r.dnd}||SdS)Nrrr ) is_nestedsizelen)r#r$Zsrc_sizeZ seq_len_posr r r! _get_seq_len s  r(cseZdZdZddddddejdddd d d ddfeeeeeeee e e ge ffe e e e eeeedd fd d Zde e e e e e e e e e e e e e e ee eee d ddZeeejefeejeje dddZddZZS)ra_A transformer model. User is able to modify the attributes as needed. The architecture is based on the paper "Attention Is All You Need". Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010. Args: d_model: the number of expected features in the encoder/decoder inputs (default=512). nhead: the number of heads in the multiheadattention models (default=8). num_encoder_layers: the number of sub-encoder-layers in the encoder (default=6). num_decoder_layers: the number of sub-decoder-layers in the decoder (default=6). dim_feedforward: the dimension of the feedforward network model (default=2048). dropout: the dropout value (default=0.1). activation: the activation function of encoder/decoder intermediate layer, can be a string ("relu" or "gelu") or a unary callable. Default: relu custom_encoder: custom encoder (default=None). custom_decoder: custom decoder (default=None). layer_norm_eps: the eps value in layer normalization components (default=1e-5). batch_first: If ``True``, then the input and output tensors are provided as (batch, seq, feature). Default: ``False`` (seq, batch, feature). norm_first: if ``True``, encoder and decoder layers will perform LayerNorms before other attention and feedforward operations, otherwise after. Default: ``False`` (after). bias: If set to ``False``, ``Linear`` and ``LayerNorm`` layers will not learn an additive bias. Default: ``True``. Examples:: >>> transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12) >>> src = torch.rand((10, 32, 512)) >>> tgt = torch.rand((20, 32, 512)) >>> out = transformer_model(src, tgt) Note: A full example to apply nn.Transformer module for the word language model is available in https://github.com/pytorch/examples/tree/master/word_language_model i皙?Nh㈵>FT)d_modelnheadnum_encoder_layersnum_decoder_layersdim_feedforwarddropout activationcustom_encodercustom_decoderlayer_norm_epsr$ norm_firstbiasrc s||d}ttjd|jj|dk r:||_n@t|||||| | | | f |}t |f| | d|}t ||||_| dk r| |_ n@t |||||| | | | f |}t |f| | d|}t ||||_ |||_||_| |_dS)Nrrtorch.nn.modules.epsr9)super__init__r_C_log_api_usage_once __class____name__encoderrrrdecoderrr_reset_parametersr.r/r$)selfr.r/r0r1r2r3r4r5r6r7r$r8r9rrfactory_kwargs encoder_layerZ encoder_norm decoder_layerZ decoder_normrBr r!r?VsB    zTransformer.__init__) r#tgtsrc_masktgt_mask memory_masksrc_key_padding_masktgt_key_padding_maskmemory_key_padding_mask src_is_causal tgt_is_causalmemory_is_causalrc  Cs|dk} |js4|d|dkr4| r4tdn&|jrZ|d|dkrZ| rZtd|d|jksz|d|jkrtd|j|||| d} |j|| ||||| | d}|S) aTake in and process masked source/target sequences. Args: src: the sequence to the encoder (required). tgt: the sequence to the decoder (required). src_mask: the additive mask for the src sequence (optional). tgt_mask: the additive mask for the tgt sequence (optional). memory_mask: the additive mask for the encoder output (optional). src_key_padding_mask: the Tensor mask for src keys per batch (optional). tgt_key_padding_mask: the Tensor mask for tgt keys per batch (optional). memory_key_padding_mask: the Tensor mask for memory keys per batch (optional). src_is_causal: If specified, applies a causal mask as ``src_mask``. Default: ``None``; try to detect a causal mask. Warning: ``src_is_causal`` provides a hint that ``src_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. tgt_is_causal: If specified, applies a causal mask as ``tgt_mask``. Default: ``None``; try to detect a causal mask. Warning: ``tgt_is_causal`` provides a hint that ``tgt_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. memory_is_causal: If specified, applies a causal mask as ``memory_mask``. Default: ``False``. Warning: ``memory_is_causal`` provides a hint that ``memory_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: - src: :math:`(S, E)` for unbatched input, :math:`(S, N, E)` if `batch_first=False` or `(N, S, E)` if `batch_first=True`. - tgt: :math:`(T, E)` for unbatched input, :math:`(T, N, E)` if `batch_first=False` or `(N, T, E)` if `batch_first=True`. - src_mask: :math:`(S, S)` or :math:`(N\cdot\text{num\_heads}, S, S)`. - tgt_mask: :math:`(T, T)` or :math:`(N\cdot\text{num\_heads}, T, T)`. - memory_mask: :math:`(T, S)`. - src_key_padding_mask: :math:`(S)` for unbatched input otherwise :math:`(N, S)`. - tgt_key_padding_mask: :math:`(T)` for unbatched input otherwise :math:`(N, T)`. - memory_key_padding_mask: :math:`(S)` for unbatched input otherwise :math:`(N, S)`. Note: [src/tgt/memory]_mask ensures that position i is allowed to attend the unmasked positions. If a BoolTensor is provided, positions with ``True`` are not allowed to attend while ``False`` values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. [src/tgt/memory]_key_padding_mask provides specified elements in the key to be ignored by the attention. If a BoolTensor is provided, the positions with the value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged. - output: :math:`(T, E)` for unbatched input, :math:`(T, N, E)` if `batch_first=False` or `(N, T, E)` if `batch_first=True`. Note: Due to the multi-head attention architecture in the transformer model, the output sequence length of a transformer is same as the input sequence (i.e. target) length of the decoder. where S is the source sequence length, T is the target sequence length, N is the batch size, E is the feature number Examples: >>> # xdoctest: +SKIP >>> output = transformer_model(src, tgt, src_mask=src_mask, tgt_mask=tgt_mask) r z-the batch number of src and tgt must be equalrz:the feature number of src and tgt must be equal to d_model)maskrP is_causalrNrOrQrRrTrU)dimr$r& RuntimeErrorr.rDrE)rGr#rLrMrNrOrPrQrRrSrTrUZ is_batchedmemoryoutputr r r!forwardys"J     zTransformer.forwardrcCst|||dS)zGenerate a square causal mask for the sequence. The masked positions are filled with float('-inf'). Unmasked positions are filled with float(0.0). r)r"rr r r!generate_square_subsequent_masks z+Transformer.generate_square_subsequent_maskcCs&|D]}|dkrt|qdS)z-Initiate parameters in the transformer model.r N) parametersr[r )rGpr r r!rFs  zTransformer._reset_parameters) NNNNNNNNF)rC __module__ __qualname____doc__Freluintrrstrrrrrboolr?r_ staticmethodrrr@_get_default_deviceget_default_dtyperr`rF __classcell__r r rKr!r2sn## [ csLeZdZdZdgZd fdd Zd eeeeeeeeddd Z Z S) raTransformerEncoder is a stack of N encoder layers. Users can build the BERT(https://arxiv.org/abs/1810.04805) model with corresponding parameters. Args: encoder_layer: an instance of the TransformerEncoderLayer() class (required). num_layers: the number of sub-encoder-layers in the encoder (required). norm: the layer normalization component (optional). enable_nested_tensor: if True, input will automatically convert to nested tensor (and convert back on output). This will improve the overall performance of TransformerEncoder when padding rate is high. Default: ``True`` (enabled). Examples:: >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8) >>> transformer_encoder = nn.TransformerEncoder(encoder_layer, num_layers=6) >>> src = torch.rand(10, 32, 512) >>> out = transformer_encoder(src) normNTcsttjd|jjt|||_||_ ||_ ||_ ||_ ||_ d}d}t|tjjsl|d}n|jr~|d}nz|jjs|dd}nb|jjs|d}nN|js|d }n<|jj|jjks|d |d }n|jjd d kr|d}|r|rtd|d|_ dS)Nr;rIz was not TransformerEncoderLayerz.norm_first was Truez#.self_attn.batch_first was not Truez2(use batch_first for better inference performance)z+.self_attn._qkv_same_embed_dim was not Truez%.activation_relu_or_gelu was not Truez.norm1.eps was not equal to z .norm2.epsrr z.self_attn.num_heads is oddzJenable_nested_tensor is True, but self.use_nested_tensor is False because F)r>r?rr@rArBrC _get_cloneslayers num_layersroenable_nested_tensoruse_nested_tensor mask_check isinstancennrr8 self_attnr$_qkv_same_embed_dimactivation_relu_or_gelunorm1r=norm2 num_headswarningswarn)rGrIrsrortrvZ enc_layerwhy_not_sparsity_fast_pathrKr r!r?s:        zTransformerEncoder.__init__)r#rXrPrYrc CsFtj|dt|d|jd}tj|ddd|jdd}|}d}|jd}|}d} d } |jj} t|d sjd } n|jsvd } n|j r| d } nv| dksd| } nZ|dkrd} nLt|dr|j rt ||sd} n&|jrd} n|dk rd} n t rd} | s||jj|jj|jjj|jjj|jj|jj|jj|jj|jj|jj|jj|jjf } ddt jjjg} t j| rzd} n<|jj | krd| } n"t !rt"dd| Drd} | s|dk rd}t j#||dd}d}t$|| }t%|||}|jD]}|||||d}q|r,|&d |'}|j(dk rB|(|}|S)!aPass the input through the encoder layers in turn. Args: src: the sequence to the encoder (required). mask: the mask for the src sequence (optional). src_key_padding_mask: the mask for the src keys per batch (optional). is_causal: If specified, applies a causal mask as ``mask``. Default: ``None``; try to detect a causal mask. Warning: ``is_causal`` provides a hint that ``mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer class. rPrXrX mask_name other_type other_name target_typeNrpFrXrrrrZ check_otherrzself.layers[0]ruz'use_nested_tensor attribute not presentz1self.use_nested_tensor (set in init) was not Truez was in training moderV3input not batched; expected src.dim() of 3 but got zsrc_key_padding_mask was NonervzImask_check enabled, and src and src_key_padding_mask was not left alignedz#NestedTensor input is not supportedz0src_key_padding_mask and mask were both suppliedautocast is enabledcpucuda'some Tensor argument has_torch_functionzsrc device is neither one of css|] }|jVqdSNZ requires_grad.0xr r r! vsz-TransformerEncoder.forward..hgrad is enabled and at least one of query or the input/output projection weights or biases requires_gradT)rv)rMrYrPg))rf_canonical_mask_none_or_dtyperrrryr$hasattrrutrainingr[rvrZ%_nested_tensor_from_mask_left_alignedZ logical_notr%is_autocast_enabledin_proj_weight in_proj_biasout_projweightr9r|r}linear1linear2utilsbackend_registration_privateuse1_backend_name overrideshas_torch_functionrtypeis_grad_enabledanyZ_nested_tensor_from_maskr(_detect_is_causal_maskZto_padded_tensorr&ro)rGr#rXrPrYr^Zconvert_to_nestedZ first_layerZsrc_key_padding_mask_for_layersrZstr_first_layerr$ tensor_args_supported_device_typeseq_lenmodr r r!r_s           zTransformerEncoder.forward)NTT)NNN rCrcrdre __constants__r?rrrjr_rnr r rKr!rs&c s\eZdZdZdgZd fdd Zd eeeeeeeeeeeeeed dd Z Z S) ra_TransformerDecoder is a stack of N decoder layers Args: decoder_layer: an instance of the TransformerDecoderLayer() class (required). num_layers: the number of sub-decoder-layers in the decoder (required). norm: the layer normalization component (optional). Examples:: >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8) >>> transformer_decoder = nn.TransformerDecoder(decoder_layer, num_layers=6) >>> memory = torch.rand(10, 32, 512) >>> tgt = torch.rand(20, 32, 512) >>> out = transformer_decoder(tgt, memory) roNcs<ttjd|jjt|||_||_ ||_ dS)Nr;) r>r?rr@rArBrCrqrrrsro)rGrJrsrorKr r!r?s   zTransformerDecoder.__init__F rLr]rNrOrQrRrTrUrc Cs`|} t||jdjj} t||| }|jD]} | | |||||||d} q*|jdk r\|| } | S)aPass the inputs (and mask) through the decoder layer in turn. Args: tgt: the sequence to the decoder (required). memory: the sequence from the last layer of the encoder (required). tgt_mask: the mask for the tgt sequence (optional). memory_mask: the mask for the memory sequence (optional). tgt_key_padding_mask: the mask for the tgt keys per batch (optional). memory_key_padding_mask: the mask for the memory keys per batch (optional). tgt_is_causal: If specified, applies a causal mask as ``tgt mask``. Default: ``None``; try to detect a causal mask. Warning: ``tgt_is_causal`` provides a hint that ``tgt_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. memory_is_causal: If specified, applies a causal mask as ``memory mask``. Default: ``False``. Warning: ``memory_is_causal`` provides a hint that ``memory_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer class. rrZN)r(rrryr$rro) rGrLr]rNrOrQrRrTrUr^rrr r r!r_s     zTransformerDecoder.forward)N)NNNNNFrr r rKr!rs" c seZdZdZdgZddejddddddf eeeee e e e ge ffee e e dd fd d Zfd d Zde ee ee e e dddZde ee ee e e dddZe e dddZZS)raxTransformerEncoderLayer is made up of self-attn and feedforward network. This standard encoder layer is based on the paper "Attention Is All You Need". Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010. Users may modify or implement in a different way during application. TransformerEncoderLayer can handle either traditional torch.tensor inputs, or Nested Tensor inputs. Derived classes are expected to similarly accept both input formats. (Not all combinations of inputs are currently supported by TransformerEncoderLayer while Nested Tensor is in prototype state.) If you are implementing a custom layer, you may derive it either from the Module or TransformerEncoderLayer class. If your custom layer supports both torch.Tensors and Nested Tensors inputs, make its implementation a derived class of TransformerEncoderLayer. If your custom Layer supports only torch.Tensor inputs, derive its implementation from Module. Args: d_model: the number of expected features in the input (required). nhead: the number of heads in the multiheadattention models (required). dim_feedforward: the dimension of the feedforward network model (default=2048). dropout: the dropout value (default=0.1). activation: the activation function of the intermediate layer, can be a string ("relu" or "gelu") or a unary callable. Default: relu layer_norm_eps: the eps value in layer normalization components (default=1e-5). batch_first: If ``True``, then the input and output tensors are provided as (batch, seq, feature). Default: ``False`` (seq, batch, feature). norm_first: if ``True``, layer norm is done prior to attention and feedforward operations, respectively. Otherwise it's done after. Default: ``False`` (after). bias: If set to ``False``, ``Linear`` and ``LayerNorm`` layers will not learn an additive bias. Default: ``True``. Examples:: >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8) >>> src = torch.rand(10, 32, 512) >>> out = encoder_layer(src) Alternatively, when ``batch_first`` is ``True``: >>> encoder_layer = nn.TransformerEncoderLayer(d_model=512, nhead=8, batch_first=True) >>> src = torch.rand(32, 10, 512) >>> out = encoder_layer(src) Fast path: forward() will use a special optimized implementation described in `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness`_ if all of the following conditions are met: - Either autograd is disabled (using ``torch.inference_mode`` or ``torch.no_grad``) or no tensor argument ``requires_grad`` - training is disabled (using ``.eval()``) - batch_first is ``True`` and the input is batched (i.e., ``src.dim() == 3``) - activation is one of: ``"relu"``, ``"gelu"``, ``torch.functional.relu``, or ``torch.functional.gelu`` - at most one of ``src_mask`` and ``src_key_padding_mask`` is passed - if src is a `NestedTensor `_, neither ``src_mask`` nor ``src_key_padding_mask`` is passed - the two ``LayerNorm`` instances have a consistent ``eps`` value (this will naturally be the case unless the caller has manually modified one without modifying the other) If the optimized implementation is in use, a `NestedTensor `_ can be passed for ``src`` to represent padding more efficiently than using a padding mask. In this case, a `NestedTensor `_ will be returned, and an additional speedup proportional to the fraction of the input that is padding can be expected. .. _`FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness`: https://arxiv.org/abs/2205.14135 r8r+r,r-FTN r.r/r2r3r4r7r$r8r9rc s| | d} tt||f|| |d| |_t||fd| i| |_t||_t||fd| i| |_||_ t |f|| d| |_ t |f|| d| |_ t||_ t||_t|trt|}|tjkst|tjjrd|_n*|tjkst|tjjr d|_nd|_||_dS)Nr:)r3r9r$r9r<r rr)r>r?r ryrrrr3rr8rr|r}dropout1dropout2rwri_get_activation_fnrfrgrrxZReLUr{geluZGELUr4 rGr.r/r2r3r4r7r$r8r9rrrHrKr r!r?#s0       z TransformerEncoderLayer.__init__cs"t|t|dstj|_dSNr4)r> __setstate__rrfrgr4rGstaterKr r!rEs  z$TransformerEncoderLayer.__setstate__)r#rMrPrYrc stj|dt|d|jd}tj|ddd|jdd}d}|dksTd |}n|jr`d }n||jjsnd }nn|jjs|d }n`|j sd }nT|j j |j j ksd}n>|j r|dk s|dk rd}n"|jjddkrd}n trd}|s0||jj|jj|jjj|jjj|j j|j j|j j|j j|jj|jj|jj|jjf }ddtjjjgtj|rXd}nFtfdd|Ds|d}n"trtdd|Drd}|s0|j |||\}}t!||jj"|jj|jj|jj|jjj|jjj|j dk|j#|j j |j j|j j|j j|j j|jj|jj|jj|jj||S|} |j#rn| |j$| | |||d} | |%| | } n0| | |j$| |||d} | | |%| } | S)aPass the input through the encoder layer. Args: src: the sequence to the encoder layer (required). src_mask: the mask for the src sequence (optional). src_key_padding_mask: the mask for the src keys per batch (optional). is_causal: If specified, applies a causal mask as ``src mask``. Default: ``False``. Warning: ``is_causal`` provides a hint that ``src_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer class. rPrMrNrpFrrVrztraining is enabledz"self_attn.batch_first was not Truez*self_attn._qkv_same_embed_dim was not Truez$activation_relu_or_gelu was not Truez#norm1.eps is not equal to norm2.epszSneither src_key_padding_mask nor src_mask are not supported with NestedTensor inputrr znum_head is oddrrrrc3s|]}|jjkVqdSr)rrrrr r!rsz2TransformerEncoderLayer.forward..z0some Tensor argument's device is neither one of css|] }|jVqdSrrrr r r!rsr)rY)&rfrrrr[rryr$rzr{r|r=r}r%r~rrrrrrr9rrrrrrrallrrZ merge_masksZ_transformer_encoder_layer_fwdZ embed_dimr8 _sa_block _ff_block) rGr#rMrPrYrrZ merged_maskZ mask_typerr rr!r_Ks   zTransformerEncoderLayer.forwardr attn_maskkey_padding_maskrYrc Cs&|j|||||d|dd}||S)NF)rr need_weightsrYrryrrGrrrrYr r r!rs z!TransformerEncoderLayer._sa_blockrrc Cs&|||||}||Sr)rr3r4rrrGrr r r!rsz!TransformerEncoderLayer._ff_block)NNF)F)rCrcrdrerrfrgrhrrrirrrjr?rrr_rrrnr r rKr!rsJH "   c seZdZdZdgZddejddddddf eeeee e e e ge ffee e e dd fd d Zfd d Zde e ee ee ee ee e e e d ddZde ee ee e e dddZde e ee ee e e dddZe e dddZZS)raTransformerDecoderLayer is made up of self-attn, multi-head-attn and feedforward network. This standard decoder layer is based on the paper "Attention Is All You Need". Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010. Users may modify or implement in a different way during application. Args: d_model: the number of expected features in the input (required). nhead: the number of heads in the multiheadattention models (required). dim_feedforward: the dimension of the feedforward network model (default=2048). dropout: the dropout value (default=0.1). activation: the activation function of the intermediate layer, can be a string ("relu" or "gelu") or a unary callable. Default: relu layer_norm_eps: the eps value in layer normalization components (default=1e-5). batch_first: If ``True``, then the input and output tensors are provided as (batch, seq, feature). Default: ``False`` (seq, batch, feature). norm_first: if ``True``, layer norm is done prior to self attention, multihead attention and feedforward operations, respectively. Otherwise it's done after. Default: ``False`` (after). bias: If set to ``False``, ``Linear`` and ``LayerNorm`` layers will not learn an additive bias. Default: ``True``. Examples:: >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8) >>> memory = torch.rand(10, 32, 512) >>> tgt = torch.rand(20, 32, 512) >>> out = decoder_layer(tgt, memory) Alternatively, when ``batch_first`` is ``True``: >>> decoder_layer = nn.TransformerDecoderLayer(d_model=512, nhead=8, batch_first=True) >>> memory = torch.rand(32, 10, 512) >>> tgt = torch.rand(32, 20, 512) >>> out = decoder_layer(tgt, memory) r8r+r,r-FTNrc s| | d} tt||f||| d| |_t||f||| d| |_t||fd| i| |_t||_t||fd| i| |_ ||_ t |f|| d| |_ t |f|| d| |_ t |f|| d| |_t||_t||_t||_t|tr t||_n||_dS)Nr:)r3r$r9r9r<)r>r?r rymultihead_attnrrrr3rr8rr|r}norm3rrdropout3rwrirr4rrKr r!r?s2          z TransformerDecoderLayer.__init__cs"d|krtj|d<t|dSr)rfrgr>rrrKr r!rs z$TransformerDecoderLayer.__setstate__rc Cs|} |jrV| ||| |||} | ||| ||||} | ||| } nJ|| || |||} || || ||||} || || } | S)agPass the inputs (and mask) through the decoder layer. Args: tgt: the sequence to the decoder layer (required). memory: the sequence from the last layer of the encoder (required). tgt_mask: the mask for the tgt sequence (optional). memory_mask: the mask for the memory sequence (optional). tgt_key_padding_mask: the mask for the tgt keys per batch (optional). memory_key_padding_mask: the mask for the memory keys per batch (optional). tgt_is_causal: If specified, applies a causal mask as ``tgt mask``. Default: ``False``. Warning: ``tgt_is_causal`` provides a hint that ``tgt_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. memory_is_causal: If specified, applies a causal mask as ``memory mask``. Default: ``False``. Warning: ``memory_is_causal`` provides a hint that ``memory_mask`` is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility. Shape: see the docs in Transformer class. )r8rr| _mha_blockr}rr) rGrLr]rNrOrQrRrTrUrr r r!r_s)zTransformerDecoderLayer.forwardrc Cs&|j||||||ddd}||SNF)rrrYrrrrr r r!rUs z!TransformerDecoderLayer._sa_block)rmemrrrYrc Cs&|j||||||ddd}||Sr)rr)rGrrrrrYr r r!r_s z"TransformerDecoderLayer._mha_blockrc Cs&|||||}||Sr)rr3r4rrrr r r!risz!TransformerDecoderLayer._ff_block)NNNNFF)F)F)rCrcrdrerrfrgrhrrrirrrjr?rrr_rrrrnr r rKr!rsf#   7  cstfddt|DS)Ncsg|]}tqSr )copydeepcopy)rimoduler r! psz_get_clones..)r range)rNr rr!rqnsrq)r4rcCs.|dkrtjS|dkrtjStd|dS)Nrgrz$activation should be relu/gelu, not )rfrgrr\)r4r r r!rss r)rXrYr&rcCsj|dk}|dkrf|dk rf|dk r$|n|d}t||j|jd}||krbt||k}nd}|S)aReturn whether the given attention mask is causal. Warning: If ``is_causal`` is not ``None``, its value will be returned as is. If a user supplies an incorrect ``is_causal`` hint, ``is_causal=False`` when the mask is in fact a causal attention.mask may lead to reduced performance relative to what would be achievable with ``is_causal=True``; ``is_causal=True`` when the mask is in fact not a causal attention.mask may lead to incorrect and unpredictable execution - in some scenarios, a causal mask may be applied based on the hint, in other execution scenarios the specified mask may be used. The choice may not appear to be deterministic, in that a number of factors like alignment, hardware SKU, etc influence the decision whether to use a mask or rely on the hint. ``size`` if not None, check whether the mask is a causal mask of the provided size Otherwise, checks for any causal mask. TNr:F)r&r"rrrjr)rXrYr&Z make_causalrZcausal_comparisonr r r!r|sr)NN)-rtypingrrrrrrrrprrfrr r4r containerr initr r3rZlinearrZ normalizationr__all__rr@rlrmrhrr"rjr(rrrrrrqrirrr r r r!sZ           6(J