U 9%e_U@sdZddlZddlZddlmZmZddlmZddlm Z m Z ddl m Z m Z ddlmZddlmZmZmZmZddlZd d lmZmZmZmZerddlmZGd d d eZd dZ ddZ!ddZ"ddZ#ddZ$ddZ%ddZ&ddZ'ddZ(dd Z)d!d"Z*d#d$Z+d%d&Z,d'd(Z-d)d*Z.d+d,Z/d-d.Z0d/d0Z1d1d2Z2d3d4Z3Gd5d6d6eZ4Gd7d8d8e5eZ6Gd9d:d:e6Z7Gd;d<dd>Z9d?d@Z:dAdBZ;dZee5e5dEdFdGZd\dLdMZ?dNdOZ@d]dPdQZAdRdSZBdTdUZCdVdWZDdXdYZEdS)^z Generic utilities N) OrderedDictUserDict)MutableMapping) ExitStackcontextmanager)fields is_dataclass)Enum)AnyContextManagerListTuple)is_flax_availableis_tf_availableis_torch_availableis_torch_fx_proxyc@seZdZdZdddZdS)cached_propertyz Descriptor that mimics @property but caches output in member variable. From tensorflow_datasets Built-in in functools from Python 3.8. NcCsX|dkr |S|jdkrtdd|jj}t||d}|dkrT||}t||||S)Nzunreadable attributeZ __cached_)fgetAttributeError__name__getattrsetattr)selfobjobjtypeattrcachedrY/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/utils/generic.py__get__-s     zcached_property.__get__)N)r __module__ __qualname____doc__r rrrrr$srcCs2|}|dkrdS|dkr dStd|dS)zConvert a string representation of truth to true (1) or false (0). True values are 'y', 'yes', 't', 'true', 'on', and '1'; false values are 'n', 'no', 'f', 'false', 'off', and '0'. Raises ValueError if 'val' is anything else. >yesony1truetr>falsenonofff0rzinvalid truth value N)lower ValueError)valrrr strtobool<s r3cCsHtt|}|drdS|dr(dS|dr6dS|drDdSd S) z Tries to guess the framework of an object `x` from its repr (brittle but will help in `is_tensor` to try the frameworks in a smart order, without the need to import the frameworks). zjs z1_get_frameworks_and_test_func..csi|]}||qSrrr>)framework_to_testrr ksz1_get_frameworks_and_test_func..)is_torch_tensor is_tf_tensor is_jax_tensoris_numpy_arrayr<appendextend)r;Z frameworksr)rBr@r_get_frameworks_and_test_funcZs rJcCsTt|}|D]}||rdSqt|r0dStrPddlm}t||rPdSdS)z Tests if `x` is a `torch.Tensor`, `tf.Tensor`, `jaxlib.xla_extension.DeviceArray` or `np.ndarray` in the order defined by `infer_framework_from_repr` Tr)TracerF)rJvaluesrrZjax.corerK isinstance)r;framework_to_test_func test_funcrKrrr is_tensorns   rPcCs t|tjSN)rMr7ndarrayr;rrr _is_numpysrTcCst|S)z/ Tests if `x` is a numpy array or not. )rTrSrrrrGsrGcCsddl}t||jSNr)torchrMTensorr;rVrrr _is_torchsrYcCsts dSt|S)z] Tests if `x` is a torch tensor or not. Safe to call even if torch is not installed. F)rrYrSrrrrDsrDcCsddl}t||jSrU)rVrMZdevicerXrrr_is_torch_devicesrZcCsts dSt|S)z] Tests if `x` is a torch device or not. Safe to call even if torch is not installed. F)rrZrSrrris_torch_devicesr[cCs8ddl}t|tr,t||r(t||}ndSt||jS)NrF)rVrMr8hasattrrZdtyperXrrr_is_torch_dtypes    r]cCsts dSt|S)z\ Tests if `x` is a torch dtype or not. Safe to call even if torch is not installed. F)rr]rSrrris_torch_dtypesr^cCsddl}t||jSrU) tensorflowrMrWr;r5rrr_is_tensorflowsracCsts dSt|S)zg Tests if `x` is a tensorflow tensor or not. Safe to call even if tensorflow is not installed. F)rrarSrrrrEsrEcCs*ddl}t|dr||St||jkS)Nris_symbolic_tensor)r_r\rbr9rWr`rrr_is_tf_symbolic_tensors  rccCsts dSt|S)z Tests if `x` is a tensorflow symbolic tensor or not (ie. not eager). Safe to call even if tensorflow is not installed. F)rrcrSrrris_tf_symbolic_tensorsrdcCsddlm}t||jSrU) jax.numpynumpyrMrR)r;jnprrr_is_jaxs rhcCsts dSt|S)zY Tests if `x` is a Jax tensor or not. Safe to call even if jax is not installed. F)rrhrSrrrrFsrFcCsddddddddd}t|ttfr>dd|DSt|ttfrZd d |DSt|}|D] \}}||rj|||Sqjt|tjr| S|Sd S) zc Convert a TensorFlow tensor, PyTorch tensor, Numpy array or python list to a python list. cSs|SrQ)detachcputolistrrrrzto_py_obj..cSs |SrQ)rfrkrlrrrrmrncSst|SrQ)r7asarrayrkrlrrrrmrncSs|SrQ)rkrlrrrrmrnr=cSsi|]\}}|t|qSr to_py_objr?kvrrrrCszto_py_obj..cSsg|] }t|qSrrp)r?orrrrAszto_py_obj..N) rMdictritemslisttuplerJr7numberrk)rZframework_to_py_objrN frameworkrOrrrrqs  rqcCsddddddddd}t|ttfr>dd|DSt|ttfrVt|St|}|D] \}}||rf|||Sqf|S) zc Convert a TensorFlow tensor, PyTorch tensor, Numpy array or python list to a Numpy array. cSs|SrQ)rirjrfrlrrrrm rnzto_numpy..cSs|SrQ)rfrlrrrrm rncSs t|SrQ)r7rorlrrrrmrncSs|SrQrrlrrrrmrnr=cSsi|]\}}|t|qSr)to_numpyrrrrrrCszto_numpy..) rMrvrrwrxryr7arrayrJ)rZframework_to_numpyrNr{rOrrrr|s r|cseZdZdZddddZfddZdd Zd d Zd d ZddZ ddZ ddZ fddZ fddZ eedddZZS) ModelOutputa Base class for all model outputs as dataclass. Has a `__getitem__` that allows indexing by integer or slice (like a tuple) or strings (like a dictionary) that will ignore the `None` attributes. Otherwise behaves like a regular python dictionary. You can't unpack a `ModelOutput` directly. Use the [`~utils.ModelOutput.to_tuple`] method to convert it to a tuple before. N)returncs4tr0ddljjjjjfdddS)zRegister subclasses as pytree nodes. This is necessary to synchronize gradients when using `torch.nn.parallel.DistributedDataParallel` with `static_graph=True` with modules that output `ModelOutput` subclasses. rNcsfjj||SrQ)utils_pytreeZ_dict_unflatten)rLcontextclsrVrrrm:rnz/ModelOutput.__init_subclass__..)rZtorch.utils._pytreerrZ_register_pytree_nodeZ _dict_flatten)rrrr__init_subclass__.s zModelOutput.__init_subclass__csBtj|||jtk}|r>t|s>t|jd|jjddS)N.za is not a dataclasss. This is a subclass of ModelOutput and so must use the @dataclass decorator.)super__init__ __class__r~r TypeErrorr!r)rargskwargsZis_modeloutput_subclassrrrr=s   zModelOutput.__init__c st}t|s"tjjdtdd|ddDsNtjjdt|dj}tfdd|ddD}|rt|st |t r| }d }n*zt |}d }Wnt k rd }YnX|rvt|D]\}}t |ttfrt|d krt |dtsB|dkr,||dj<ntd |d qt|d|d|ddk r|d|d<qn|dk r||dj<n,|D]&}t|j} | dk r| |j<qdS)zeCheck the ModelOutput dataclass. Only occurs if @dataclass decorator has been used. z has no fields.css|]}|jdkVqdSrQ)defaultr?fieldrrr Vsz,ModelOutput.__post_init__..rNz. should not have more than one required field.rc3s|]}t|jdkVqdSrQ)rnamerrrrrZsTFzCannot set key/value for z&. It needs to be a tuple (key, value).)rlenr1rrallrrrPrMrvrwiterr enumeraterxryr8r) rZ class_fieldsZ first_fieldZother_fields_are_noneiteratorZfirst_field_iteratoridxelementrrtrrr __post_init__LsN           zModelOutput.__post_init__cOstd|jjddS)Nz$You cannot use ``__delitem__`` on a instance. Exceptionrrrrrrrr __delitem__szModelOutput.__delitem__cOstd|jjddS)Nz#You cannot use ``setdefault`` on a rrrrrr setdefaultszModelOutput.setdefaultcOstd|jjddS)NzYou cannot use ``pop`` on a rrrrrrpopszModelOutput.popcOstd|jjddS)NzYou cannot use ``update`` on a rrrrrrupdateszModelOutput.updatecCs.t|trt|}||S||SdSrQ)rMr8rvrwto_tuple)rrsZ inner_dictrrr __getitem__s  zModelOutput.__getitem__cs4||kr"|dk r"t||t||dSrQ)keysr __setitem__ __setattr__)rrvaluerrrrszModelOutput.__setattr__cs t||t||dSrQ)rrr)rkeyrrrrrszModelOutput.__setitem__cstfddDS)za Convert self to a tuple containing all the attributes/keys that are not `None`. c3s|]}|VqdSrQr)r?rsrrrrsz'ModelOutput.to_tuple..)ryrrrrrrszModelOutput.to_tuple)rr!r"r#rrrrrrrrrrr r r __classcell__rrrrr~ s  8  r~c@seZdZdZeddZdS) ExplicitEnumzC Enum with more explicit error message for missing values. cCs(t|d|jdt|jdS)Nz is not a valid z, please select one of )r1rrx_value2member_map_r)rrrrr _missing_szExplicitEnum._missing_N)rr!r"r# classmethodrrrrrrsrc@seZdZdZdZdZdZdS)PaddingStrategyz Possible values for the `padding` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. longest max_lengthZ do_not_padN)rr!r"r#ZLONGESTZ MAX_LENGTHZ DO_NOT_PADrrrrrsrc@s eZdZdZdZdZdZdZdS) TensorTypez Possible values for the `return_tensors` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. r4r5r7r6N)rr!r"r#ZPYTORCHZ TENSORFLOWZNUMPYZJAXrrrrrs rc@s2eZdZdZeedddZddZddZd S) ContextManagersz Wrapper for `contextlib.ExitStack` which enters a collection of context managers. Adaptation of `ContextManagers` in the `fastcore` library. )context_managerscCs||_t|_dSrQ)rrstack)rrrrrrszContextManagers.__init__cCs|jD]}|j|qdSrQ)rr enter_context)rZcontext_managerrrr __enter__s zContextManagers.__enter__cOs|jj||dSrQ)r__exit__rrrrrszContextManagers.__exit__N) rr!r"r#r r rrrrrrrrsrcCsnt|}|dkrt|j}n"|dkr4t|j}n t|j}|jD]"}|dkrF|j|jdkrFdSqFdS)zr Check if a given model can return loss. Args: model_class (`type`): The class of the model. r5r4Z return_lossTF)infer_frameworkinspect signaturecallforward__call__ parametersr) model_classr{rprrrcan_return_losss  rcCsr|j}t|}|dkr$t|j}n"|dkr:t|j}n t|j}d|kr^dd|jDSdd|jDSdS)zq Find the labels used by a given model. Args: model_class (`type`): The class of the model. r5r4ZQuestionAnsweringcSs g|]}d|ks|dkr|qS)label)Zstart_positionsZ end_positionsrr?rrrrrAszfind_labels..cSsg|]}d|kr|qS)rrrrrrrAsN)rrrrrrrr)rZ model_namer{rrrr find_labelss rr)d parent_key delimitercCsddd}t||||S)z/Flatten a nested dict into a single level dict.rrcssd|D]V\}}|r(t||t|n|}|rTt|trTt|||dEdHq||fVqdS)N)r)rwr8rMr flatten_dict)rrrrsrtrrrr _flatten_dicts z#flatten_dict.._flatten_dict)rr)rv)rrrrrrrr s rF) use_temp_dirc cs*|r t }|VW5QRXn|VdSrQ)tempfileTemporaryDirectory)Z working_dirrZtmp_dirrrrworking_or_temp_dirs rcCst|rtj||dSt|r6|dkr,|jS|j|St|rTddl}|j||dSt|rjt j||dSt dt |ddS)z Framework-agnostic version of `numpy.transpose` that will work on torch/TensorFlow/Jax tensors as well as NumPy arrays. )axesNr)permz"Type not supported for transpose: r) rGr7 transposerDTZpermuterEr_rFrgr1r9)r}rr5rrrr"srcCsnt|rt||St|r&|j|St|rBddl}|||St|rVt||Stdt |ddS)z Framework-agnostic version of `numpy.reshape` that will work on torch/TensorFlow/Jax tensors as well as NumPy arrays. rNz Type not supported for reshape: r) rGr7reshaperDrEr_rFrgr1r9)r}Znewshaper5rrrr5s    rcCst|rtj||dSt|r:|dkr.|S|j|dSt|rXddl}|j||dSt|rntj||dStdt |ddS)z Framework-agnostic version of `numpy.squeeze` that will work on torch/TensorFlow/Jax tensors as well as NumPy arrays. axisNdimrz Type not supported for squeeze: r) rGr7squeezerDrEr_rFrgr1r9r}rr5rrrrHsrcCstt|rt||St|r(|j|dSt|rFddl}|j||dSt|r\tj||dSt dt |ddS)z Framework-agnostic version of `numpy.expand_dims` that will work on torch/TensorFlow/Jax tensors as well as NumPy arrays. rrNr$Type not supported for expand_dims: r) rGr7 expand_dimsrDZ unsqueezerEr_rFrgr1r9rrrrr[s  rcCsbt|rt|St|r"|St|r.Nr)rwrMryrx)Zauto_maprrrrrradd_model_info_to_auto_maps rcCst|D]l}|j}|j}|ds6|ds6|dkr<dS|dsN|dkrTdS|dsp|d sp|d kr dSq td |d d S)z Infers the framework of a given model without using isinstance(), because we cannot guarantee that the relevant classes are imported or available. r_ZkerasZTFPreTrainedModelr5rVZPreTrainedModelr4Zflaxr6ZFlaxPreTrainedModelz%Could not infer framework from class rN)rgetmror!rr:r)rZ base_classmodulerrrrrsr)rr)F)N)N)Fr#rr collectionsrrcollections.abcr contextlibrr dataclassesrrenumr typingr r r r rfr7Z import_utilsrrrrrergpropertyrr3r<rJrPrTrGrYrDrZr[r]r^rarErcrdrhrFrqr|r~r8rrrrrrrboolrrrrrrrrrrrrsf