U ,È-e®.ã@söddlZddlmZmZmZmZmZmZddlZddl m Z ddl m Z m Z ddlmZddlmZmZejgZe e¡Ze  e  ej¡j¡Zee  d¡kZee  d ¡kZee  d ¡kZee  d ¡kZee  d ¡kZd d „Z d%ej!ej"e#ej!dœdd„Z$Gdd„dej%ƒZ&d&e&ej"e#e&dœdd„Z'd'eej!e&fej"ee#eej!e&fdœdd„Z(edej)fe#e#ej)dœdd„Z*ee#e#e#ee#eee#ej"fdœdd„Z+ddœeej)eej)fee,eej)dfdœd d!„Z-ej)eej.e#e#fd"œd#d$„Z/dS)(éN)ÚCallableÚListÚOptionalÚSetÚTupleÚUnion)Úversion)Ú storage_ptrÚ storage_size)Únné)Úis_torch_tpu_availableÚloggingz2.0z1.12z1.11z1.8.0cCs6ddlm}tr ||||j|ƒS||||j|jƒSdS)z  A function that calls the internal `_softmax_backward_data` PyTorch method and that adjusts the arguments according to the torch version detected. r)Ú_softmax_backward_dataN)ÚtorchrÚis_torch_less_than_1_11ÚdimZdtype)ÚparentZ grad_outputÚoutputrÚselfr©rú[/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/pytorch_utils.pyÚsoftmax_backward_data&s r)ÚlayerÚindexrÚreturncCsè| |jj¡}|j ||¡ ¡ ¡}|jdk rX|dkrF|j ¡ ¡}n|j| ¡ ¡}t|j ¡ƒ}t |ƒ||<t j |d|d|jdk d |jj¡}d|j_ |j  | ¡¡d|j_ |jdk räd|j_ |j  | ¡¡d|j_ |S)a  Prune a linear layer to keep only entries in index. Used to remove heads. Args: layer (`torch.nn.Linear`): The layer to prune. index (`torch.LongTensor`): The indices to keep in the layer. dim (`int`, *optional*, defaults to 0): The dimension on which to keep the indices. Returns: `torch.nn.Linear`: The pruned layer as a new layer with `requires_grad=True`. Nr r)ÚbiasFT)ÚtoÚweightÚdeviceÚ index_selectÚcloneÚdetachrÚlistÚsizeÚlenr ÚLinearÚ requires_gradÚcopy_Ú contiguous©rrrÚWÚbÚnew_sizeZ new_layerrrrÚprune_linear_layer4s"  ( r.cs(eZdZdZ‡fdd„Zdd„Z‡ZS)ÚConv1Da 1D-convolutional layer as defined by Radford et al. for OpenAI GPT (and also used in GPT-2). Basically works like a linear layer but the weights are transposed. Args: nf (`int`): The number of output features. nx (`int`): The number of input features. csLtƒ ¡||_t t ||¡¡|_t t |¡¡|_ tj j |jdddS)Ng{®Gáz”?)Zstd) ÚsuperÚ__init__Únfr Ú ParameterrÚemptyrZzerosrÚinitZnormal_)rr2Únx©Ú __class__rrr1as  zConv1D.__init__c CsF| ¡dd…|jf}t |j| d| d¡¡|j¡}| |¡}|S)Néÿÿÿÿ)r$r2rZaddmmrÚviewr)rÚxZsize_outrrrÚforwardhs  zConv1D.forward)Ú__name__Ú __module__Ú __qualname__Ú__doc__r1r<Ú __classcell__rrr7rr/Vs r/cCsÈ| |jj¡}|j ||¡ ¡ ¡}|dkr<|j ¡ ¡}n|j| ¡ ¡}t|j ¡ƒ}t |ƒ||<t |d|dƒ |jj¡}d|j_ |j  |  ¡¡d|j_ d|j_ |j  |  ¡¡d|j_ |S)a Prune a Conv1D layer to keep only entries in index. A Conv1D work as a Linear layer (see e.g. BERT) but the weights are transposed. Used to remove heads. Args: layer ([`~pytorch_utils.Conv1D`]): The layer to prune. index (`torch.LongTensor`): The indices to keep in the layer. dim (`int`, *optional*, defaults to 1): The dimension on which to keep the indices. Returns: [`~pytorch_utils.Conv1D`]: The pruned layer as a new layer with `requires_grad=True`. rr FT)rrrr r!r"rr#r$r%r/r'r(r)r*rrrÚprune_conv1d_layeros rBcCs^t|tjƒr&t|||dkrdn|dSt|tƒrJt|||dkrBdn|dStd|j›ƒ‚dS)aÇ Prune a Conv1D or linear layer to keep only entries in index. Used to remove heads. Args: layer (`Union[torch.nn.Linear, Conv1D]`): The layer to prune. index (`torch.LongTensor`): The indices to keep in the layer. dim (`int`, *optional*): The dimension on which to keep the indices. Returns: `torch.nn.Linear` or [`~pytorch_utils.Conv1D`]: The pruned layer as a new layer with `requires_grad=True`. Nr©rr zCan't prune layer of class )Ú isinstancer r&r.r/rBÚ ValueErrorr8)rrrrrrÚ prune_layers   rF.)Ú forward_fnÚ chunk_sizeÚ chunk_dimrc s(t|ƒdkst|›dƒ‚tt ˆ¡jƒ}|t|ƒkrPtd|›dt|ƒ›dƒ‚|dkr |djˆ}|D],}|jˆ|krltd|›d|jˆ›ƒ‚ql|djˆ|dkrÎtd|djˆ›d |›ƒ‚|djˆ|‰t‡‡fd d „|Dƒƒ}t‡fd d „t|ŽDƒƒ}t j |ˆd Sˆ|ŽS)aZ This function chunks the `input_tensors` into smaller input tensor parts of size `chunk_size` over the dimension `chunk_dim`. It then applies a layer `forward_fn` to each chunk independently to save memory. If the `forward_fn` is independent across the `chunk_dim` this function will yield the same result as directly applying `forward_fn` to `input_tensors`. Args: forward_fn (`Callable[..., torch.Tensor]`): The forward function of the model. chunk_size (`int`): The chunk size of a chunked tensor: `num_chunks = len(input_tensors[0]) / chunk_size`. chunk_dim (`int`): The dimension over which the `input_tensors` should be chunked. input_tensors (`Tuple[torch.Tensor]`): The input tensors of `forward_fn` which will be chunked Returns: `torch.Tensor`: A tensor with the same shape as the `forward_fn` would have given if applied`. Examples: ```python # rename the usual forward() fn to forward_chunk() def forward_chunk(self, hidden_states): hidden_states = self.decoder(hidden_states) return hidden_states # implement a chunked forward function def forward(self, hidden_states): return apply_chunking_to_forward(self.forward_chunk, self.chunk_size_lm_head, self.seq_len_dim, hidden_states) ```rz" has to be a tuple/list of tensorszforward_chunk_fn expects z arguments, but only z input tensors are givenz/All input tenors have to be of the same shape: z, found shape zThe dimension to be chunked z( has to be a multiple of the chunk size c3s|]}|jˆˆdVqdS)rCN)Úchunk)Ú.0Ú input_tensor)rIÚ num_chunksrrÚ êsz,apply_chunking_to_forward..c3s|]}ˆ|ŽVqdS)Nr)rKZinput_tensors_chunk)rGrrrNìsrC) r%ÚAssertionErrorÚinspectÚ signatureÚ parametersrEÚshapeÚtupleÚziprÚcat) rGrHrIZ input_tensorsZnum_args_in_forward_chunk_fnZ tensor_shaperLZinput_tensors_chunksZ output_chunksr)rIrGrMrÚapply_chunking_to_forward¨s,& ÿ ÿÿrW)ÚheadsÚn_headsÚ head_sizeÚalready_pruned_headsrcsvt ||¡}t|ƒ|}|D]&‰ˆt‡fdd„|Dƒƒ‰d|ˆ<q| d¡ ¡ d¡}t t|ƒ¡|  ¡}||fS)a3 Finds the heads and their indices taking `already_pruned_heads` into account. Args: heads (`List[int]`): List of the indices of heads to prune. n_heads (`int`): The number of heads in the model. head_size (`int`): The size of each head. already_pruned_heads (`Set[int]`): A set of already pruned heads. Returns: `Tuple[Set[int], torch.LongTensor]`: A tuple with the indices of heads to prune taking `already_pruned_heads` into account and the indices of rows/columns to keep in the layer weight. c3s|]}|ˆkrdndVqdS)r rNr)rKÚh©ÚheadrrrNsz3find_pruneable_heads_and_indices..rr9r ) rZonesÚsetÚsumr:r)ÚeqZaranger%Úlong)rXrYrZr[Úmaskrrr]rÚ find_pruneable_heads_and_indicesós   rd)Úindexing)ÚtensorsrercGstj|d|iŽS)z· Wrapper around torch.meshgrid to avoid warning messages about the introduced `indexing` argument. Reference: https://pytorch.org/docs/1.13/generated/torch.meshgrid.html re)rÚmeshgrid)rerfrrrrgsrg)ÚtensorrcCs@|jjdkr(tƒr(ddl}|j |¡}nt|ƒ}|j|t|ƒfS)a“ Unique identifier to a tensor storage. Multiple different tensors can share the same underlying storage. For example, "meta" tensors all share the same storage, and thus their identifier will all be equal. This identifier is guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with non-overlapping lifetimes may have the same id. ZxlarN)rÚtyper Ú torch_xlaZ_XLACZ_xla_get_tensor_idr r )rhrjÚ unique_idrrrÚid_tensor_storages rl)r)r )N)0rPÚtypingrrrrrrrÚ packagingrZsafetensors.torchr r r Úutilsr rZ LayerNormZALL_LAYERNORM_LAYERSZ get_loggerr=ÚloggerÚparseÚ __version__Ú base_versionZparsed_torch_version_baseZ"is_torch_greater_or_equal_than_2_0Z#is_torch_greater_or_equal_than_1_12Z#is_torch_greater_or_equal_than_1_11rZis_torch_1_8_0rr&Z LongTensorÚintr.ÚModuler/rBrFZTensorrWrdÚstrrgrrlrrrrÚsV    ""ÿ  þ  þ Lþ ÿ þ