U 9%e)@sddlmZmZddlmZddlmZmZmZerHddl m Z m Z m Z GdddeZ deeeedd d Zee ed d d ZeeeddddZdS))c_floatsizeof)Enum) TYPE_CHECKINGOptionalUnionAutoFeatureExtractor AutoProcessor AutoTokenizerc@s"eZdZeZeedddZdS)ParameterFormat)returncCs t|jS)zf Number of byte required for this data type Returns: Integer > 0 )rvalue)selfrV/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/onnx/utils.pysizeszParameterFormat.sizeN)__name__ __module__ __qualname__rFloatpropertyintrrrrrr sr ) dimensionfixed_dimensionnum_token_to_addrcCs|dkr |}||8}|S)ze Args: dimension: fixed_dimension: num_token_to_add: Returns: rr)rrrrrr compute_effective_axis_dimension&s r)num_parametersdtypercCs ||jS)a7 Compute the size taken by all the parameters in the given the storage format when serializing the model Args: num_parameters: Number of parameters to be saved dtype: The data format each parameter will be saved Returns: Size (in byte) taken to save all the parameters )r)rrrrr"compute_serialized_parameters_size9s r )r r r ) model_namerc Csddlm}m}m}z ||WStttfk rd\}}z||}Wnttfk rdYnXz||}Wnttfk rYnX|dk r|dk rtd|dn.|dkr|dkrYdS|dk r|YS|YSYnXdS)a> Gets a preprocessor (tokenizer, feature extractor or processor) that is available for `model_name`. Args: model_name (`str`): Name of the model for which a preprocessor are loaded. Returns: `Optional[Union[AutoTokenizer, AutoFeatureExtractor, AutoProcessor]]`: If a processor is found, it is returned. Otherwise, if a tokenizer or a feature extractor exists, it is returned. If both a tokenizer and a feature extractor exist, an error is raised. The function returns `None` if no preprocessor is found. rr )NNNz&Couldn't auto-detect preprocessor for z1. Found both a tokenizer and a feature extractor.)r r r Zfrom_pretrained ValueErrorOSErrorKeyError)r!r r r Z tokenizerZfeature_extractorrrrget_preprocessorGs,  r&N)r)ctypesrrenumrtypingrrrr"r r r r rrr strr&rrrrs