U ,-e0@sdZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZeeZeeejZd d d d ZGd dde Zeeje_ejjdk rejjjddddej_dS)z9 Processing saving/loading class for common processors. N)Path)OptionalUnion)custom_object_save)PreTrainedTokenizerBase)PushToHubMixin copy_funcdirect_transformers_importloggingrZFeatureExtractionMixinZImageProcessingMixin)Z AutoTokenizerZAutoFeatureExtractorZAutoImageProcessorc @seZdZdZddgZdZdZdZddZddZ de d d d Z e de eejfee eejfe e ee ee fedddZe dddZe ddZeddZdS)ProcessorMixinza This is a mixin used to provide saving/loading functionality for all processor classes. Zfeature_extractorZ tokenizerNc Os0|D]}||jkrtd|dqt||jD]*\}}||krRtd|dq0|||<q0t|t|jkrtdt|jdd|jdt|d|D]\}}t||d }t ||}t |t rt d d |D}n tt |}t ||std t |jd |d|dt|||qdS)NzUnexpected keyword argument .z!Got multiple values for argument zThis processor requires z arguments: z, z. Got z arguments instead._classcss |]}|dk rtt|VqdSNgetattrtransformers_module.0nr^/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/processing_utils.py Msz*ProcessorMixin.__init__..z Received a z for argument z, but a z was expected.) attributes TypeErrorziplen ValueErrorjoinitemsrAUTO_TO_BASE_CLASS_MAPPINGget isinstancetuplertype__name__setattr)selfargskwargskeyargattribute_name class_nameZ proper_classrrr__init__6s,  (    zProcessorMixin.__init__cs0fddjD}d|}jjd|S)Nc s&g|]}d|dtt|qS)z- z: )reprr)rnamer'rr Ysz+ProcessorMixin.__repr__.. z: )rr __class__r%)r'Zattributes_reprrr1r__repr__Xs zProcessorMixin.__repr__F) push_to_hubc  s^|dd}|dk r@tdt|dddk r8td||d<tj|dd|r|dd}|d |tj j d }j |f|} |}j dk rʇfd d jD}d d |D} t|| djD]0} t| } t| dr| jj| |qЈj dk r:jD]$} t| } t| tr| jd=q|rZj|||||dddS)a Saves the attributes of this processor (feature extractor, tokenizer...) in the specified directory so that it can be reloaded using the [`~ProcessorMixin.from_pretrained`] method. This class method is simply calling [`~feature_extraction_utils.FeatureExtractionMixin.save_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`]. Please refer to the docstrings of the methods above for more information. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file and the tokenizer files will be saved (directory will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`Dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. use_auth_tokenNVThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.tokenV`token` and `use_auth_token` are both specified. Please set only the argument `token`.T)exist_okcommit_messagerepo_idcsg|]}t|qSr)r)rr,r1rrr2sz2ProcessorMixin.save_pretrained..cSs g|]}t|tr|jn|qSr)r"r init_kwargs)rarrrr2s)config_set_processor_classZauto_map)r<r9)popwarningswarn FutureWarningr!rosmakedirssplitpathsepZ _create_repoZ_get_files_timestamps _auto_classrrrhasattrrBr4r%save_pretrainedr"rr?Z_upload_modified_files) r'Zsave_directoryr6r)r7r<r=Zfiles_timestampsattrsZconfigsr, attributerr1rrN]sN             zProcessorMixin.save_pretrainedmain)pretrained_model_name_or_path cache_dirforce_downloadlocal_files_onlyr9revisionc Ksz||d<||d<||d<||d<|dd}|dk rTtdt|dk rPtd|}|dk rd||d <|j|f|} || S) a Instantiate a processor associated with a pretrained model. This class method is simply calling the feature extractor [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`], image processor [`~image_processing_utils.ImageProcessingMixin`] and the tokenizer [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] methods. Please refer to the docstrings of the methods above for more information. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on huggingface.co. Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a user or organization name, like `dbmdz/bert-base-german-cased`. - a path to a *directory* containing a feature extractor file saved using the [`~SequenceFeatureExtractor.save_pretrained`] method, e.g., `./my_model_directory/`. - a path or url to a saved feature extractor JSON *file*, e.g., `./my_model_directory/preprocessor_config.json`. **kwargs Additional keyword arguments passed along to both [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] and [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`]. rSrTrUrVr7Nr8r:r9)rCrDrErFr_get_arguments_from_pretrained) clsrRrSrTrUr9rVr)r7r(rrrfrom_pretraineds&( zProcessorMixin.from_pretrained AutoProcessorcCsDt|ts|j}ddlmm}t||s:t|d||_dS)a Register this class with a given auto class. This should only be used for custom feature extractors as the ones in the library are already mapped with `AutoProcessor`. This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoProcessor"`): The auto class to register this new feature extractor with. rNz is not a valid auto class.) r"strr%Ztransformers.models.automodelsautorMrrL)rXZ auto_classZ auto_modulerrrregister_for_auto_classs   z&ProcessorMixin.register_for_auto_classc Ksg}|jD]~}t||d}t|trjtdd|D}|dd}|r`|ddk r`|d}qt|d}n tt|}||j|f|q |S)Nrcss$|]}|dk rtt|ndVqdSrrrrrrrsz@ProcessorMixin._get_arguments_from_pretrained..use_fastTrr)rrr"r#r!rappendrY) rXrRr)r(r,r-classesr_Zattribute_classrrrrWs      z-ProcessorMixin._get_arguments_from_pretrainedcCst||jd}t|ddS)Nrmodel_input_names)rr)r'Zfirst_attributerrrrbsz ProcessorMixin.model_input_names)F)NFFNrQ)rZ)r% __module__ __qualname____doc__rZfeature_extractor_classZtokenizer_classrLr.r5boolrN classmethodrr[rGPathLikerrYr^rWpropertyrbrrrrr *s8"J =  r processorrZzprocessor files)objectZ object_classZ object_files)rerGrDpathlibrtypingrrZdynamic_module_utilsrZtokenization_utils_baserutilsrr r r Z get_loggerr%logger__file__parentrr r r6formatrrrrs,    n