U ,-ej@s^dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddlm Z m Z mZmZmZddlmZmZmZmZmZmZmZeeZddZeeejfd d d Zeeejfeed d dZ eeejfeed ddZ!eeejfeedddZ"eeejfeedddZ#eeeejfej$dddZ%d(eeejfeeeeejfe&e&ee eefeee&efeee&eeeeed ddZ'd)eeeejfeeeejfe&e&ee eefeee&efeee&eeeeej$d ddZ(d*e eeejfee eed d!d"Z)d#d$Z*d%Z+d&d'Z,dS)+z3Utilities to dynamically load objects from the Hub.N)Path)AnyDictListOptionalUnion)HF_MODULES_CACHE TRANSFORMERS_DYNAMIC_MODULE_NAME cached_fileextract_commit_hashis_offline_modeloggingtry_to_load_from_cachecCsPttjkrdStjttjtddttd}|sL|t dS)z_ Creates the cache directory for modules with an init, and adds it to the Python path. NTexist_ok __init__.py) r syspathappendosmakedirsrexiststouch importlibinvalidate_caches) init_pathrb/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/dynamic_module_utils.pyinit_hf_modules*s   r)namecCs\ttt|}|js*t|jtj|dd|d}|sX| t dS)z Creates a dynamic module in the cache directory for modules. Args: name (`str` or `os.PathLike`): The name of the dynamic module to create. TrrN) rrr resolveparentrcreate_dynamic_modulerrrrr)r Zdynamic_module_pathrrrrr#:s  r#) module_filereturnc CsVt|ddd}|}W5QRXtjd|tjd}|tjd|tjd7}tt|S)z Get the list of modules that are relatively imported in a module file. Args: module_file (`str` or `os.PathLike`): The module file to inspect. Returns: `List[str]`: The list of relative imports in the module. rutf-8encodingz^\s*import\s+\.(\S+)\s*$flagsz^\s*from\s+\.(\S+)\s+import)openreadrefindall MULTILINElistset)r$fcontentZrelative_importsrrrget_relative_importsPs r5csd}|g}g|sg}|D]}|t|qt|jfdd|D}fdd|D}dd|D}t|dk}|qS)a Get the list of all files that are needed for a given module. Note that this function recurses through the relative imports (if a imports b and b imports c, it will return module files for b and c). Args: module_file (`str` or `os.PathLike`): The module file to inspect. Returns: `List[str]`: The list of all relative imports a given module needs (recursively), which will give us the list of module files a given module needs. Fcsg|]}t|qSr)str).0m) module_pathrr |sz-get_relative_import_files..csg|]}|kr|qSrrr7r3)all_relative_importsrrr:}scSsg|]}|dqS).pyrr;rrrr:~sr)extendr5rr"len)r$Z no_changeZfiles_to_checkZ new_importsr3Znew_import_filesr)r<r9rget_relative_import_fileses    r@)filenamer%c Cs~t|ddd}|}W5QRXtjdd|tjtjBd}tjd|tjd}|tjd|tjd7}d d |D}tt|S) a Extracts all the libraries (not relative imports this time) that are imported in a file. Args: filename (`str` or `os.PathLike`): The module file to inspect. Returns: `List[str]`: The list of all packages required to use the input module. r&r'r(z \s*try\s*:\s*.*?\s*except\s*.*?:r*z^\s*import\s+(\S+)\s*$z^\s*from\s+(\S+)\s+importcSs$g|]}|ds|ddqS).r) startswithsplit)r7imprrrr:s zget_imports..) r,r-r.subr0DOTALLr/r1r2)rAr3r4importsrrr get_importss rJc Cszt|}g}|D]2}zt|Wqtk r@||YqXqt|dkrrtdd|dd|dt|S)a. Check if the current Python environment contains all the libraries that are imported in a file. Will raise if a library is missing. Args: filename (`str` or `os.PathLike`): The module file to check. Returns: `List[str]`: The list of relative imports in the file. rz\This modeling file requires the following packages that were not found in your environment: z, z. Run `pip install  `)rJr import_module ImportErrorrr?joinr5)rArIZmissing_packagesrFrrr check_importss  rP) class_namer9r%cCs$|tjjd}t|}t||S)a, Import a module on the cache directory for modules and extract a class from it. Args: class_name (`str`): The name of the class to import. module_path (`str` or `os.PathLike`): The path to the module to import. Returns: `typing.Type`: The class looked for. rC)replacerrseprrMgetattr)rQr9modulerrrget_class_in_modules  rVF) pretrained_model_name_or_pathr$ cache_dirforce_downloadresume_downloadproxiestokenrevisionlocal_files_only repo_type _commit_hashr%c  Ks| dd} | dk r4tdt|dk r0td| }trL|sLtdd}t|}t j |} | rrt j |}n"| dt j j}t|||| | d}g}z8t|||||||||| | d }| s||kr||Wn.tk rtd |d |d YnXt|}tt j j|}t|tt|}|t j |kr||rbt|t||szt|||t|D]Z}|d }t j ||}||rt|t||s~t|||tq~nt!|| }||}|t j j|}t|||s,t|||t|D]L}||d s0t"||d ||||||||d ||d q0t#|dkr|dkrd dd|D}| dkrdn| d}d||}t$d|d|dt j ||S)a Prepares Downloads a module from a local folder or a distant repo and returns its path inside the cached Transformers module. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. module_file (`str`): The name of the module file containing the class to look for. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. repo_type (`str`, *optional*): Specify the repo type (useful when downloading from a space for instance). Passing `token=True` is required when you want to use a private model. Returns: `str`: The path to the module inside the cache. use_auth_tokenNVThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.z+Offline mode: forcing local_files_only=TrueT/)rXr]r_) rXrYr[rZr^r\r]r_r`zCould not locate the z inside rCr=)rXrYrZr[r\r]r^r`r cSsg|]}d|qS)z- rr;rrrr:nsz*get_cached_module_file..rBzs/zhttps://huggingface.co/z9A new version of the following files was downloaded from z: z . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.)%popwarningswarn FutureWarning ValueErrorr loggerinfor6rrisdirbasenamerRrSrr rEnvironmentErrorerrorrPr r#rr rfilecmpcmpshutilcopyrrrOr get_cached_module_filer?warning)rWr$rXrYrZr[r\r]r^r_r`Zdeprecated_kwargsrais_local submoduleZ cached_moduleZ new_filesZresolved_module_fileZmodules_neededZfull_submoduleZsubmodule_pathZ module_neededZmodule_needed_fileZ commit_hashZ repo_type_strurlrrrrus>           ru) class_referencerWrXrYrZr[r\r]r^r_ code_revisionr%c  Ks| dd} | dk r4tdt|dk r0td| }d|krL|d\} }n|} |d\}}| dkrr|| krr|} t| |d|||||| || d }t||dd S) a Extracts a class from a module file, present in the local folder or repository of a model. Calling this function will execute the code in the module file found locally or downloaded from the Hub. It should therefore only be called on trusted repos. Args: class_reference (`str`): The full name of the class to load, including its module and optionally its repo. pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. This is used when `class_reference` does not specify another repo. module_file (`str`): The name of the module file containing the class to look for. class_name (`str`): The name of the class to import in the module. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download (`bool`, *optional*, defaults to `False`): Whether or not to delete incompletely received file. Attempts to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `huggingface-cli login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. repo_type (`str`, *optional*): Specify the repo type (useful when downloading from a space for instance). code_revision (`str`, *optional*, defaults to `"main"`): The specific revision to use for the code on the Hub, if the code leaves in a different repository than the rest of the model. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. Passing `token=True` is required when you want to use a private model. Returns: `typing.Type`: The class, dynamically imported from the module. Examples: ```python # Download module `modeling.py` from huggingface.co and cache then extract the class `MyBertModel` from this # module. cls = get_class_from_dynamic_module("modeling.MyBertModel", "sgugger/my-bert-model") # Download module `modeling.py` from a given repo and cache then extract the class `MyBertModel` from this # module. cls = get_class_from_dynamic_module("sgugger/my-bert-model--modeling.MyBertModel", "sgugger/another-bert-model") ```raNrbrcz--rCr=)rXrYrZr[r\r]r^r_rB) rfrgrhrirjrErurVrR)rzrWrXrYrZr[r\r]r^r_r{kwargsraZrepo_idr$rQZ final_modulerrrget_class_from_dynamic_modulezs8X  r})objfolderconfigr%c sjdkr&tdd|ddSfdd}t|ttfrT|D] }||qDn|dk rd||g}tjjj}t |t |j }t ||| |t|D],}t |t |j }t ||| |q|S)a  Save the modeling files corresponding to a custom model/configuration/tokenizer etc. in a given folder. Optionally adds the proper fields in a config. Args: obj (`Any`): The object for which to save the module files. folder (`str` or `os.PathLike`): The folder where to save. config (`PretrainedConfig` or dictionary, `optional`): A config in which to register the auto_map corresponding to this custom object. Returns: `List[str]`: The list of files saved. __main__z We can't save the code defining z in z as it's been defined in __main__. You should put this code in a separate module so we can include it in the saved folder and make it easier to share via the Hub.Nc s jj}|dd}|djj}d|krd}d}jjdr|djj}tdddk rtd}|j}|dd}|d|j}n|djj}||f}t|tr|di} || j <| |d<n*t|dddk r||j j <n j |i|_ dS)NrC TokenizerZFastslow_tokenizer_classauto_map) __class__ __module__rE__name__endswithrT isinstancedictgetZ _auto_classr) _config module_nameZ last_moduleZ full_namerZfast_tokenizer_classZslow_tokenizerZslow_tok_module_nameZlast_slow_tok_modulerr~rr_set_auto_map_in_config s,     z3custom_object_save.._set_auto_map_in_config)rrkrvrr1tuplermodules__file__rr rsrtrr@) r~rrrcfgresultZ object_file dest_fileZ needed_filerrrcustom_object_saves*        rcCs tddS)NzLoading this model requires you to execute custom code contained in the model repository on your localmachine. Please set the option `trust_remote_code=True` to permit loading of this model.)rj)signumframerrr_raise_timeout_error@src Cs|dkr|rd}n|rtdkrzhttjttt|dkrztd|d|d}|dkrhd}q8|dkr8d}q8tdWqtk rtd|d|d YqXn|rtdd|r|s|std |d |S) NFrzThe repository for z contains custom code which must be executed to correctlyload the model. You can inspect the repository content at https://hf.co/z. You can avoid this prompt in future by passing the argument `trust_remote_code=True`. Do you wish to run the custom code? [y/N] )yesy1T)non0rBzS. Please pass the argument `trust_remote_code=True` to allow custom code to be run.zLoading z requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option `trust_remote_code=True` to remove this error.) TIME_OUT_REMOTE_CODEsignalSIGALRMralarminputlower Exceptionrj)Ztrust_remote_codeZ model_nameZhas_local_codeZhas_remote_codeZanswerrrrresolve_trust_remote_codeJs6        r) NFFNNNFNN) NFFNNNFNN)N)-__doc__rqrrr.rsrrtypingrgpathlibrrrrrrutilsr r r r r rrZ get_loggerrrkrr6PathLiker#r5r@rJrPTyperVboolrur}rrrrrrrrs $ !  3  z(L