U 9%eŠ@s@dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZd d lmZd d lmZd d l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*d d l+m,Z,m-Z-m.Z.e./e0Z1dZ2dZ3dZ4dZ5e!d7Z!eeeedZ6de2iZ7e-e!Gddde&Z8dS)z Tokenization classes for fast tokenizers (provided by HuggingFace's tokenizers library). For slow (python) tokenizers see tokenization_utils.py N) defaultdict)AnyDictListOptionalTupleUnion)Encoding) Tokenizer)Decoder) BpeTrainerUnigramTrainerWordLevelTrainerWordPieceTrainer)convert_slow_tokenizer)PreTrainedTokenizer) INIT_TOKENIZER_DOCSTRING AddedToken BatchEncodingPreTokenizedInputPreTokenizedInputPairPreTrainedTokenizerBaseSpecialTokensMixin TextInput TextInputPairTruncationStrategy)PaddingStrategyadd_end_docstringsloggingztokenizer.jsonzspecial_tokens_map.jsonztokenizer_config.jsonzadded_tokens.jsonu tokenizer_object ([`tokenizers.Tokenizer`]): A [`tokenizers.Tokenizer`] object from 🤗 tokenizers to instantiate from. See [Using tokenizers from 🤗 tokenizers](../fast_tokenizers) for more information. tokenizer_file ([`str`]): A path to a local JSON file representing a previously serialized [`tokenizers.Tokenizer`] object from 🤗 tokenizers. )BPEUnigram WordLevel WordPiecetokenizer_filecseZdZUdZeZdZeed<fddZ e e dddZ e e dd d Z e edd d Zeeefdd dZe eeefdddZe eeefdddZe eeefdddZeeefdddZedddZe edddZe edddZdKeee ee e e e e e eeee fe!efd d d!Z"e#ee!efe#ee!efd"d#d$Z$eed%d&d'Z%eeed(d)d*Z&dLe!e#eefed+d,d-Z'dMe ed.d/d0Z(dNe#ee!efe e#ee!efd1d2d3Z)dOeeee e!ed4d5d6Z*e+e,eeeed7d8d9Z-de+j.e,j/dd:ddddddddddfe#e!e0e!e1e!e2e!e3fe e+e,eeee eeeeee ee e e e e e e4d;dd?d@Z6e!eed"dAdBZ7dPe#ee!efe e edCdDdEZ8dQe#ee9j:feeee eeeedFdGdHZ;dRdIdJZ<Z=S)SPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classc s|dd}|dd}|dd}|dd}|dd}|rZ|dkrZ|jdkrZtd|dk rnt|}nR|dk r|st|}n:|dk rt|}n(|jdk r|j||}t|}ntd||_|dk r| |j d|_ |jj } | dk rB|jj f| |d | d |d | d |d | d |d | dn |j|jj} | dk r|jjf| |d| d|d| d|d| d |d | d|d| dtjf||s|j|jdddS)Ntokenizer_objectZ__slow_tokenizerr$ from_slowF slow_to_fastzCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.a-Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofT)special_tokens)popr& ValueErrorcopydeepcopy TokenizerFast from_filer _tokenizerupdate init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__ _add_tokensZall_special_tokens_extended) selfargskwargsr'Zslow_tokenizerZfast_tokenizer_filer(r)Zfast_tokenizer _truncation_padding __class__c/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/tokenization_utils_fast.pyrH^sX                z PreTrainedTokenizerFast.__init__)returncCsdS)NTrQrJrQrQrRis_fastszPreTrainedTokenizerFast.is_fastcCsdS)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. TrQrTrQrQrRcan_save_slow_tokenizersz/PreTrainedTokenizerFast.can_save_slow_tokenizercCs|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). FZwith_added_tokensr=Zget_vocab_sizerTrQrQrR vocab_sizesz"PreTrainedTokenizerFast.vocab_sizecCs|jjddS)NTrW)r= get_vocabrTrQrQrRrZsz!PreTrainedTokenizerFast.get_vocabcCs|SN)rZrTrQrQrRvocabszPreTrainedTokenizerFast.vocabcCs ddt|jdddDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cSsi|]\}}|j|qSrQcontent.0vkrQrQrR sz@PreTrainedTokenizerFast.added_tokens_encoder..cSs|dSNrrQitemrQrQrRz>PreTrainedTokenizerFast.added_tokens_encoder..keysortedadded_tokens_decoderitemsrTrQrQrRadded_tokens_encodersz,PreTrainedTokenizerFast.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )r=Zget_added_tokens_decoderrTrQrQrRrmsz,PreTrainedTokenizerFast.added_tokens_decodercCs ddt|jdddDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. cSsi|]\}}|j|qSrQr]r_rQrQrRrcsz;PreTrainedTokenizerFast.get_added_vocab..cSs|dSrdrQrerQrQrRrgrhz9PreTrainedTokenizerFast.get_added_vocab..rirkrTrQrQrRget_added_vocabsz'PreTrainedTokenizerFast.get_added_vocabcCs|jjddS)zD Size of the full vocabulary with the added tokens. TrWrXrTrQrQrR__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )r=rTrQrQrRbackend_tokenizersz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )r=decoderrTrQrQrRrsszPreTrainedTokenizerFast.decoderFT) encodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboserSc Cs|dkrd|jk}|dkr$d|jk}|r@|jdk r@|g|j} n|g} tt} | D]|} | d| j|rz| d| j|r| d| j|r| d| j|r| d| j |rR| dt | jqR| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) NZtoken_type_idsattention_mask input_idsspecial_tokens_maskZoffset_mappingr4) Zmodel_input_namesZ overflowingrlistappendidsZtype_idsr|r~offsetslen) rJrtrurvrwrxryrzr{ encodingsZ encoding_dicterQrQrR_convert_encodings*  z)PreTrainedTokenizerFast._convert_encoding)tokensrScs2|dkr dSt|tr |Sfdd|DS)aT Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. Ncsg|]}|qSrQ)#_convert_token_to_id_with_added_vocr`tokenrTrQrR +szAPreTrainedTokenizerFast.convert_tokens_to_ids..) isinstancestrrrJrrQrTrRconvert_tokens_to_idss   z-PreTrainedTokenizerFast.convert_tokens_to_ids)rrScCs|j|}|dkr|jS|Sr[)r= token_to_idZ unk_token_id)rJrindexrQrQrRr-s z;PreTrainedTokenizerFast._convert_token_to_id_with_added_voc)rrScCs|jt|Sr[)r= id_to_tokenint)rJrrQrQrR_convert_id_to_token3sz,PreTrainedTokenizerFast._convert_id_to_token) new_tokensrScCs|r|j|S|j|Sr[)r=add_special_tokensZ add_tokens)rJrr6rQrQrRrI6s z#PreTrainedTokenizerFast._add_tokens)pairrScCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )r=num_special_tokens_to_add)rJrrQrQrRr<sz1PreTrainedTokenizerFast.num_special_tokens_to_add)rskip_special_tokensrScCsRt|tr|j|Sg}|D].}t|}|r:||jkr:q||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rrr=rZall_special_idsr)rJrrrrrQrQrRconvert_ids_to_tokensQs  z-PreTrainedTokenizerFast.convert_ids_to_tokens)textrrrScKs|jf|||d|S)N)r text_pairr)Z encode_plusr)rJrrrrLrQrQrRtokenizeksz PreTrainedTokenizerFast.tokenizepadding_strategyr.r*r-r5c s|jj|jj}|tjkr.dk rv|jnH|||j|jd}dkrNd}nfdd|D}||krv|jjf||t j kr|dk r|j nB|t j kr|nd} | |j |j|j|j|d}||kr|jjf|dS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). N)r*r-r/r,csi|]}||dqSr[get)r`rbrMrQrRrcszFPreTrainedTokenizerFast.set_truncation_and_padding..)r4r,Zpad_idr0r2r5)r=rArErDO_NOT_TRUNCATErDvaluer+rBr DO_NOT_PADZ no_paddingZ MAX_LENGTHr3Z pad_token_idr0r1rF) rJrr.r*r-r5rNtargetcurrentr4rQrrRset_truncation_and_paddingns8     z2PreTrainedTokenizerFast.set_truncation_and_paddingr)batch_text_or_text_pairsrrr.r*r-is_split_into_wordsr5return_tensorsrurvrwrxryrzr{rScst|ttfs"tdt|dj|||||djj|||d}fdd|D}i}|ddD]fdd|D}||<q|d d|D}rg}t |D]"\}\}}||gt |d 7}q||d <|d D]} ||qt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got )r)rZis_pretokenizedc s&g|]}j|dqS))rtrurvrwrxryrzr{)r)r`rt)rvrzryrwrxrurJr{rQrRrs z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|]\}}|D]}|qqSrQrQ)r`rf_rrirQrRrs cSsg|]\}}|D]}|qqSrQrQ)r`rrfrrQrQrRrsr}overflow_to_sample_mapping)Z tensor_type) rtupler TypeErrortyperr=Z encode_batchkeys enumerater&_eventual_warn_about_too_long_sequencer)rJrrrr.r*r-rr5rrurvrwrxryrzr{rZtokens_and_encodingsZsanitized_tokensstackZsanitized_encodingsritoksrr}rQ) rjrvrzryrwrxrurJr{rR_batch_encode_pluss@    z*PreTrainedTokenizerFast._batch_encode_plus)rrrrr.r*r-rr5rrurvrwrxryrzr{rScKs|r||fgn|g}|j|f||||||| | | | | ||||d|}| dkrl| sltdd|D|j}||d|||S)N)rrrr.r*r-r5rrurvrwrxryrzr{cSs8i|]0\}}|t|dkr0t|dtr0|dn|qS)r)rrr)r`rjrrQrQrRrc8sz8PreTrainedTokenizerFast._encode_plus..r})rrrnrr)rJrrrrr.r*r-rr5rrurvrwrxryrzr{rLZ batched_inputZbatched_outputrQrQrR _encode_plus s> z$PreTrainedTokenizerFast._encode_pluscCs|jj|Sr[)rrrsdecoderrQrQrRconvert_tokens_to_stringCsz0PreTrainedTokenizerFast.convert_tokens_to_string) token_idsrclean_up_tokenization_spacesrScKsZ|dd|_t|tr|g}|jj||d}|dk r:|n|j}|rR||}|S|SdS)NZuse_source_tokenizerF)r)r7r@rrr=rrZclean_up_tokenization)rJrrrrLrZ clean_textrQrQrR_decodeFs  zPreTrainedTokenizerFast._decode)save_directory file_names legacy_formatfilename_prefixrSc Cst|}|jdkr"|dkr"td|dks2|dko@|jdk o@|j}|dkpP|dk}|rtj||rj|dndt}|}|rt |ddd &} t j |d ddd d } | | W5QRX|j ||d } || |f}|rtj||r|dndt} |j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-wzutf-8)rt)indent sort_keys ensure_ascii )r)rr&r8rVospathjoinADDED_TOKENS_FILErpopenjsondumpswriteZsave_vocabularyTOKENIZER_FILErrsave) rJrrrrZ save_slowZ save_fastZadded_tokens_fileZ added_vocabfZout_strZ vocab_filesr$rQrQrR_save_pretrained^s>   z(PreTrainedTokenizerFast._save_pretrainedc st|j}|d}|d} d} |dddkrRi|dd<g|dd<n|ddd kr|dd dk r|dd } |dd| d } dk r| kr| } d |dd <| d gg|dd<n6|ddd kri|dd<ntd|ddddk rBd|dkrB|ddkrB|dd|dd<tt|g} |D]r} | dd}| dd}|ddd kr|sqZdk r| dkr| d| d<| t f| qZ|dk r| ||dddkr d|kr |dddk r |dd|d<|dddkr^d|kr^|dddk r^|dd|d<|ddd kr| dk r| |d<|ddk r|dddkrt j |d<t|dd}|f|| d|}j|||d| dk rt}d| krp| dD]`}| d|d}dk r>fdd|D}|| d|d<fdd|D| d|d <qd!D]N}|| krt| |\}}dk r|kr|}|}||g| |<qt| |d<tt||j}tj}|d"|D]}t|d#|dk rt||}dk r>|kr>|}t|d#|}t|t r|t ||j|j|j|jd$d%||<n|||<q|j}|dk r| |t|d kr||d"<|jfd&i|S)'uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrr r\Zmergesr!unk_idrg)r"r#z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenspecialidr^Zcontinuing_subword_prefixZend_of_word_suffixZ pre_tokenizer ByteLevelZinitial_alphabet)rYr6)r4trainerr6rcsg|]}||qSrQrr)special_tokens_maprQrRrszCPreTrainedTokenizerFast.train_new_from_iterator..csg|]}|qSrQ)rr) tokenizerrQrRrsr)clssepadditional_special_tokensrT) single_wordlstriprstrip normalizedrr') rloadsr=Zto_strr7r8r;Zfrom_strrrrextendpre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGZtrain_from_iteratorrr?r9rZSPECIAL_TOKENS_ATTRIBUTESremovegetattrrrrrrrrrP)rJZ text_iteratorrYr4Znew_special_tokensrrLZtokenizer_jsonrrrrr6Z added_tokenrrZ trainer_classrZtrained_tokenizer_jsonrjrZ special_tokenrZtoken_idZspecial_tokens_listZspecial_token_fullrrQ)rrrRtrain_new_from_iterators"               "           z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)F)F)NF)FN)NN)NNN)>__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesr&r__annotations__rHpropertyboolrUrVrrYrrrZr\rorrmrprqr;rr DecoderFastrs EncodingFastrrrrrrrrrrIrrrrrrrrrrrrrrrrrrPathLikerr __classcell__rQrQrOrRr%Ms2    D    /(  L X  9   4r%)9rr9rr collectionsrtypingrrrrrrZtokenizers.pre_tokenizersZpre_tokenizersrZ tokenizersr rr r;Ztokenizers.decodersr rZtokenizers.trainersr r rrrZtokenization_utilsrZtokenization_utils_baserrrrrrrrrrutilsrrrZ get_loggerrloggerrZSPECIAL_TOKENS_MAP_FILEZTOKENIZER_CONFIG_FILErrrr%rQrQrQrRs8        0