U 9%eyN@sdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl ZddlmZddlmZddlmZerdd lmZdd lmZeeZd d iZd d dddddiZddddddZdZGdddeZdS)z! Tokenization class for model T5.N)copyfile) TYPE_CHECKINGAnyDictListOptionalTuple)import_protobuf)PreTrainedTokenizer) AddedToken) TextInput)logging vocab_filez spiece.modelz9https://huggingface.co/t5-small/resolve/main/spiece.modelz8https://huggingface.co/t5-base/resolve/main/spiece.modelz9https://huggingface.co/t5-large/resolve/main/spiece.modelz6https://huggingface.co/t5-3b/resolve/main/spiece.modelz7https://huggingface.co/t5-11b/resolve/main/spiece.model)zt5-smallzt5-basezt5-largezt5-3bzt5-11biu▁cseZdZdZeZeZeZ ddgZ d9e e e efdd fd d Zd:d dZeddZeddZddZd;eee eeeeedfdd ZddZddZeeeedddZdd(ee d)fd*d+ Z!ed,d-Z"d.d/Z#d0d1Z$d2d3Z%d4d5Z&d?e e e e'e d6d7d8Z(Z)S)@ T5Tokenizera Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 100): Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. legacy (`bool`, *optional*): Whether or not the `legacy` behaviour of the tokenizer should be used. Legacy is before the merge of #24622 and #25224 which includes fixes to properly handle tokens that appear after special tokens. A simple example: - `legacy=True`: ```python >>> from transformers import T5Tokenizer >>> tokenizer = T5Tokenizer.from_pretrained("t5-base", legacy=True) >>> tokenizer.encode("Hello .") [8774, 32099, 3, 5, 1] ``` - `legacy=False`: ```python >>> from transformers import T5Tokenizer >>> tokenizer = T5Tokenizer.from_pretrained("t5-base", legacy=False) >>> tokenizer.encode("Hello .") # the extra space `[3]` is no longer here [8774, 32099, 5, 1] ``` Checkout the [pull request](https://github.com/huggingface/transformers/pull/24565) for more details. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). Z input_idsZattention_maskdN)sp_model_kwargsreturnc svt|ddd}t|ddd}t|ddd}|dkr6in||_||_||_tjf|j|_|j||dk rdd|D} |dkr|t| krt d|d|dnd dt |D} | }i|_ t t| D]6} td | d ddddd |j t|jd || <q|dkr(t d|jdd}||_|| dd|_||_||_tjf||||||j|d| dS)NT)rstriplstripcSsg|]}dt|kr|qS) s z(T5Tokenizer.__init__..rzBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokenscSsg|]}d|dqS)r>rrirrrrsrr )Z single_wordrrspecialz2You are using the default legacy behaviour of the a`. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 from_slowF) eos_token unk_token pad_token extra_idsadditional_special_tokensrlegacy)r rrZ _extra_idsspmSentencePieceProcessorsp_modelLoadlen ValueErrorrangeZ_added_tokens_decoderloggerZ warning_once __class__r+get_spm_processorpopsuper__init__) selfrr&r'r(r)r*rr+kwargsZ extra_tokensr"r4rrr8sZ      zT5Tokenizer.__init__Fc Cstjf|j}|js|r(||j|St|jdZ}|}td|j j d}|j |}| }d|_|j||}||W5QRX|S)NrbzThe new behaviour of z (with `self.legacy = False`)F)r,r-rr+r/ropenreadr r4__name__Z ModelProtoZ FromStringZNormalizerSpecZadd_dummy_prefixnormalizer_specZ MergeFromZSerializeToStringZLoadFromSerializedProto)r9r%Z tokenizerfr.Z model_pb2modelr@rrrr5s    zT5Tokenizer.get_spm_processorc CsZ|tjkrVtj|}|dk r(||kr(|S|dkrVtd|d|d|d|d t|S)NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5. For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`. - Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding. - If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding. - To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)rmax_model_input_sizeswarningswarn FutureWarning)Zpretrained_model_name_or_pathZmax_model_lengthZinit_max_model_lengthZdeprecated_max_model_lengthrrr!_eventually_correct_t5_max_lengths    z-T5Tokenizer._eventually_correct_t5_max_lengthcCs |jSN)r.Zget_piece_sizer9rrr vocab_sizeszT5Tokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokensr!rIrr sz)T5Tokenizer.get_vocab..)r2rJupdateZadded_tokens_encoder)r9ZvocabrrIr get_vocabs zT5Tokenizer.get_vocab) token_ids_0 token_ids_1already_has_special_tokensrcsZ|rtj||ddS|dkr2dgt|dgSdgt|dgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rNrOrPNrr$)r7get_special_tokens_maskr0)r9rNrOrPr;rrrQsz#T5Tokenizer.get_special_tokens_maskcCstttdd|jS)NcSsttd|dk S)Nz)boolresearch)rrrr$z1T5Tokenizer.get_sentinel_tokens..)listsetfilterr*rIrrrget_sentinel_tokens"szT5Tokenizer.get_sentinel_tokenscsfddDS)Ncsg|]}|qSr)Zconvert_tokens_to_ids)rtokenrIrrr(sz6T5Tokenizer.get_sentinel_token_ids..)rZrIrrIrget_sentinel_token_ids'sz"T5Tokenizer.get_sentinel_token_ids) token_idsrcCsBt|dkr2|d|jkr2td|jd|S||jgSdS)z.Do not add eos again if user already added it.rzThis sequence already has zQ. In future versions this behavior may lead to duplicated eos tokens being added.N)r0 eos_token_idrDrEr&)r9r]rrr_add_eos_if_not_present*s  z#T5Tokenizer._add_eos_if_not_present)rNrOrcCs<|jg}|dkr"t||dgSt||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)r_r0)r9rNrOZeosrrr$create_token_type_ids_from_sequences5sz0T5Tokenizer.create_token_type_ids_from_sequencescCs,||}|dkr|S||}||SdS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r`)r9rNrOrrr build_inputs_with_special_tokensKs   z,T5Tokenizer.build_inputs_with_special_tokenscCs|j}d|d<|S)Nr.)__dict__copy)r9staterrr __getstate__es zT5Tokenizer.__getstate__cCs8||_t|dsi|_tjf|j|_|j|jdS)Nr)rchasattrrr,r-r.r/r)r9drrr __setstate__js  zT5Tokenizer.__setstate__r )textrc st|jst|dkr"tj|f|Stjt|tdf|}t|dkrp|dtkrp|d|jkrp|dd}|S)z Converts a string to a list of tokens. If `self.legacy` is set to `False`, a prefix token is added unless the first token is special. r r$N)r+r0r7tokenizeSPIECE_UNDERLINEreplaceall_special_tokens)r9rjZadd_special_tokensr:tokensr;rrrlus & zT5Tokenizer.tokenizecCst|jt|jSrH)r0r.encoderr'rIrrrunk_token_lengthszT5Tokenizer.unk_token_lengthcKs^|jj|td}|js$|tdfs(|S|jj|j|td}t||jkrZ||jdS|S)u( Returns a tokenized string. We de-activated the `add_dummy_prefix` option, thus the sentencepiece internals will always strip any SPIECE_UNDERLINE. For example: `self.sp_model.encode(f"{SPIECE_UNDERLINE}Hey", out_type = str)` will give `['H', 'e', 'y']` instead of `['▁He', 'y']`. Thus we always encode `f"{unk_token}text"` and strip the `unk_token`. Here is an example with `unk_token = ""` and `unk_token_length = 4`. `self.tokenizer.sp_model.encode(" Hey", out_type = str)[4:]`. )Zout_typerkN) r.rqrr+ startswithrmr'r0rr)r9rjr:rprrr _tokenizes zT5Tokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r.Z piece_to_id)r9r[rrr_convert_token_to_idsz T5Tokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r.Z IdToPiece)r9indexr[rrr_convert_id_to_tokens z T5Tokenizer._convert_id_to_tokencCsg}|dt|d<d}d}|D]F}||jkrZ|s<|d7}||j||7}d}g}q"||d}q"||j|7}|S)z:Converts a sequence of tokens (string) in a single string.rFrkT)rrmror.decodeappendstrip)r9rpZcurrent_sub_tokensZ out_stringZprev_is_specialr[rrrconvert_tokens_to_strings  z$T5Tokenizer.convert_tokens_to_string)save_directoryfilename_prefixrc Cstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|n8tj |jst |d}|j }||W5QRX|fS)NzVocabulary path (z) should be a directory-rxrwb)ospathisdirr3errorjoinVOCAB_FILES_NAMESabspathrisfilerr=r.Zserialized_model_protowrite)r9r}r~Zout_vocab_filefiZcontent_spiece_modelrrrsave_vocabularys (  zT5Tokenizer.save_vocabulary)rrrrNNN)F)NF)N)N)F)N)*r? __module__ __qualname____doc__rZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESrCZmodel_input_namesrrrrr8r5 staticmethodrGpropertyrJrMrintrRrQrZr\r`rarbrfrirlrrrtrurwr|rr __classcell__rrr;rr?snKF           r) rrrSrDshutilrtypingrrrrrrZ sentencepiecer,Zconvert_slow_tokenizerr Ztokenization_utilsr Ztokenization_utils_baser r utilsrZ get_loggerr?r3rrrrmrrrrrs<