U 9%ec%@sddlmZmZmZddlZddlZddlm Z e e Z eej ejfeedddZdej eeeeej dd d Zdd dZd ddZej ej dddZd!ej eeddddZddZddZddZdS)")ListOptionalUnionN)logging)tensorreturncsTt|tjrt|jSt||jtdkr4S|j}fddt|DS)z Deal with dynamic shape in tensorflow cleanly. Args: tensor (`tf.Tensor` or `np.ndarray`): The tensor we want the shape of. Returns: `List[int]`: The shape of the tensor as a list. Ncs$g|]\}}|dkr|n|qSN).0isZdynamicr T/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/tf_utils.py .szshape_list..) isinstancenpndarraylistshapetfZ TensorShapeas_list enumerate)rZstaticr rr shape_lists    r)logitsaxisnamercCstjj|d||dS)a Stable wrapper that returns the same output as `tf.nn.softmax`, but that works reliably with XLA on CPU. It is meant as a workaround for the [following issue](https://github.com/tensorflow/tensorflow/issues/55682), and will be removed after it gets fixed. The arguments and outputs are the same as `tf.nn.softmax`, and relies on the fact that `softmax(x) = softmax(x + c)` (see https://ogunlao.github.io/2020/04/26/you_dont_really_know_softmax.html). Args: logits (`tf.Tensor`): Must be one of the following types: half, float32, float64. axis (`int`, *optional*): The dimension softmax would be performed on. The default is -1 which indicates the last dimension. name (`str`, *optional*): A name for the operation. Returns: `tf.Tensor`: A Tensor. Has the same type and shape as logits. g& .>rrr)rnnZsoftmaxrr r rstable_softmax1srh㈵>c Cs|jjdks"|jjdks"t|ts*tdtjj||gdd\}}|dkrdg|jj}t||||<t ||}t ||}tjj ||||||d}|S)NrzOOnly 1D weight and bias tensors are supported for now, with only a single axis.T)ZaxesZkeepdimsr!)offsetscaleZvariance_epsilon) rrankrintNotImplementedErrorrrZmomentsrreshapeZbatch_normalization) inputsweightZbiasepsilonrZmeanZvarianceroutputsr r rfunctional_layernormIs""  r,cCs|dkr||jj7}|dkr(||jj7}||kr4|St|}tj|||d}tj|d||g||ddgdd}t||S)Nrrr)rr$rmathZ reduce_prodconcatr')inputZ start_dimZend_dimZin_shapeZ flattened_dimZ out_shaper r rflattenis   *r1)encoder_attention_maskrcCst|tjst|}|jjdkr>|dddddddf}|jjdkrb|ddddddf}td|j||jj}|S)z Invert an attention mask (e.g., switches 0. and 1.). Args: encoder_attention_mask (`torch.Tensor`): An attention mask. Returns: `tf.Tensor`: The inverted attention mask. Nr) rrTensorZconvert_to_tensorrr$castdtypemin)r2Zencoder_extended_attention_maskr r rinvert_attention_mask{s    r9 input_ids)r embed_dim tensor_namerc Cs>tjj|tj||jdd|dtj|d|dddS)a `tf.gather`, on which TF embedding layers are based, won't check positive out of bound indices on GPU, returning zeros instead. This function adds a check against that dangerous silent behavior. Args: tensor (`tf.Tensor`): The tensor of indices to check. embed_dim (`int`): The embedding dimension. tensor_name (`str`, *optional*): The name of the tensor to use in the error message. )r7zThe maximum value of z (z>) must be smaller than the embedding layer's input dimension (z9). The likely cause is some problem at tokenization time.)messageN)r debuggingZ assert_lessr6r7r.Z reduce_max)rr;r<r r rcheck_embeddings_within_boundss r?c sdfdd|D}|r.tdd|t|}d}t||}tfdd|Drt|d7}t||}qH|dkrt|D]\}}||jd ||f<qn ||j|<d S) aSaves attributes (data) of the specified name into the HDF5 group. This method deals with an inherent problem of HDF5 file which is not able to store data larger than HDF5_OBJECT_HEADER_LIMIT bytes. Args: group: A pointer to a HDF5 group. name: A name of the attributes to save. data: Attributes data to store. Raises: RuntimeError: If any single attribute is too large to be saved. Copied from Keras to Transformers to avoid versioning issues. icsg|]}t|kr|qSr )lenr xZHDF5_OBJECT_HEADER_LIMITr rrs z1save_attributes_to_hdf5_group..zSThe following attributes cannot be saved to HDF5 file because they are larger than z bytes: rc3s|]}|jkVqdSr )nbytesrArCr r sz0save_attributes_to_hdf5_group..%s%dN) RuntimeErrorrZasarrayZ array_splitanyrattrs) grouprdataZbad_attributesZdata_npyZ num_chunksZ chunked_datachunk_idZ chunk_datar rCrsave_attributes_to_hdf5_groups   rMcCsj||jkr dd|j|D}nFg}d}d||f|jkrf|dd|jd||fD|d7}q(|S)aLoads attributes of the specified name from the HDF5 group. This method deals with an inherent problem of HDF5 file which is not able to store data larger than HDF5_OBJECT_HEADER_LIMIT bytes. Args: group: A pointer to a HDF5 group. name: A name of the attributes to load. Returns: data: Attributes data. Copied from Keras to Transformers to avoid versioning issues. cSs$g|]}t|dr|dn|qSdecodeutf8hasattrrOr nr r rrsz3load_attributes_from_hdf5_group..rrFcSs$g|]}t|dr|dn|qSrNrQrSr r rrsr)rIextend)rJrrKrLr r rload_attributes_from_hdf5_groups  rVcCsdd}tj||S)zwExpands 1-dimensional `Tensor`s into 2-dimensional `Tensor`s. Copied from Keras to here to avoid versioning issues.cSs*t|tjr&|jjdkr&tj|ddS|S)Nrr!r-)rrr5rr$Z expand_dims)tr r r_expand_single_1d_tensorsz+expand_1d.._expand_single_1d_tensor)rnestZ map_structure)rKrXr r r expand_1dsrZ)NN)r r!)rr!)r:)typingrrrnumpyrZ tensorflowrutilsrZ get_logger__name__loggerr5rr%rstrrr,r1r9r?rMrVrZr r r rs   " /