U ,-eM@s@dZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZmZmZmZmZmZmZmZmZmZdd lm Z m!Z!m"Z"m#Z#e#$e%Z&e e'e'fZ(e e(Z)e*Z+e e+Z,d d d d Z-dddddddddd Z.dddZ/dZ0eddZ1ddZ2GdddeZ3dS)zTokenization classes for LUKE.N)Mapping) lru_cache)DictListOptionalTupleUnion)PreTrainedTokenizer) ENCODE_KWARGS_DOCSTRING AddedToken BatchEncoding EncodedInputPaddingStrategy TensorType TextInput TextInputPairTruncationStrategy to_py_obj)add_end_docstrings is_tf_tensoris_torch_tensorloggingz vocab.jsonz merges.txtzentity_vocab.json) vocab_file merges_fileentity_vocab_filezEhttps://huggingface.co/studio-ousia/luke-base/resolve/main/vocab.jsonzFhttps://huggingface.co/studio-ousia/luke-large/resolve/main/vocab.json)zstudio-ousia/luke-basezstudio-ousia/luke-largezEhttps://huggingface.co/studio-ousia/luke-base/resolve/main/merges.txtzFhttps://huggingface.co/studio-ousia/luke-large/resolve/main/merges.txtzLhttps://huggingface.co/studio-ousia/luke-base/resolve/main/entity_vocab.jsonzMhttps://huggingface.co/studio-ousia/luke-large/resolve/main/entity_vocab.jsoniav return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **entity_ids** -- List of entity ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **entity_position_ids** -- List of entity positions in the input sequence to be fed to a model. - **entity_token_type_ids** -- List of entity token type ids to be fed to a model (when `return_token_type_ids=True` or if *"entity_token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **entity_attention_mask** -- List of indices specifying which entities should be attended to by the model (when `return_attention_mask=True` or if *"entity_attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **entity_start_positions** -- List of the start positions of entities in the word token sequence (when `task="entity_span_classification"`). - **entity_end_positions** -- List of the end positions of entities in the word token sequence (when `task="entity_span_classification"`). - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) cCstttdtddtttdtddtttdtdd}|dd}d }td D],}||krf|||d ||d7}qfd d |D}tt||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNrcSsg|] }t|qS)chr).0nr$r$k/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/luke/tokenization_luke.py sz$bytes_to_unicode..)listrangeordappenddictzip)bscsr'br$r$r(bytes_to_unicodes L    r3cCs6t}|d}|ddD]}|||f|}q|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairsZ prev_charcharr$r$r( get_pairss r9cseZdZdZeZeZeZ ddgZ dNfdd Z e ddZ ddZddZddZddZd d!Zd"d#ZdOeeeeeeed$d%d&ZdPeeeeeeeed'fd(d) ZdQeeeeeeed$d*d+ZdRd,d-ZeeedSee ee feee ee feee!ee!feee!ee!feee"ee"feee"ee"feeee#e$feee#e%feeeeeeeeeeee#e&feeeeeeeeee'd0d1d2Z(dddddd.e$j)e%j*ddd/dddddddddd.fee eee ee!ee!ee"ee"ee$e%eeeeeeeeeeee#e&feeeeeeeeee'd3d4d5Z+ddd.e$j)e%j*ddd/dddddddddd.feee ee,feeee!ee-e!e!ffeeee"ee-e"e"ffee$e%eeeeeeeeeeee#e&feeeeeeeeee'd6d7d8Z.ee"ee!d9d:d;Z/dTee eee ee"ee"ee!ee!e-e0e0e0e0e0e0fd<d=d>Z1eeed.e$j)e%j*ddd/dddddddd.fee-eedfee-eeeeeefee-eee-eefeee-eeffee$e%eeeeeeeee#eeeeeeeee'd?d@dAZ2eeedUeeeeeeeeeeeeee-eefeee-eefeeee#e$feee#e%feeeeeeeeee#e&feeeeeeeeeee'dBdCdDZ3dVee'ee'e4e#e5fe4e#ee5fee4e#e5ffeee#e$feeeeeeeeeee#e&fee'dE dFdGZ6dde$j)ddfee4e#e5fe'feeeee$eeeee7dHdIdJZ8dWe#ee#e-e#dKdLdMZ9Z:S)X LukeTokenizera" Constructs a LUKE tokenizer, derived from the GPT-2 tokenizer, using byte-level Byte-Pair-Encoding. This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from transformers import LukeTokenizer >>> tokenizer = LukeTokenizer.from_pretrained("studio-ousia/luke-base") >>> tokenizer("Hello world")["input_ids"] [0, 31414, 232, 2] >>> tokenizer(" Hello world")["input_ids"] [0, 20920, 232, 2] ``` You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance. When used with `is_split_into_words=True`, this tokenizer will add a space before each word (even the first one). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. It also creates entity sequences, namely `entity_ids`, `entity_attention_mask`, `entity_token_type_ids`, and `entity_position_ids` to be used by the LUKE model. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. entity_vocab_file (`str`): Path to the entity vocabulary file. task (`str`, *optional*): Task for which you want to prepare sequences. One of `"entity_classification"`, `"entity_pair_classification"`, or `"entity_span_classification"`. If you specify this argument, the entity sequence is automatically created based on the given entity span(s). max_entity_length (`int`, *optional*, defaults to 32): The maximum length of `entity_ids`. max_mention_length (`int`, *optional*, defaults to 30): The maximum number of tokens inside an entity span. entity_token_1 (`str`, *optional*, defaults to ``): The special token used to represent an entity span in a word token sequence. This token is only used when `task` is set to `"entity_classification"` or `"entity_pair_classification"`. entity_token_2 (`str`, *optional*, defaults to ``): The special token used to represent an entity span in a word token sequence. This token is only used when `task` is set to `"entity_pair_classification"`. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. cls_token (`str`, *optional*, defaults to `""`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. mask_token (`str`, *optional*, defaults to `""`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (LUKE tokenizer detect beginning of words by the preceding space). input_idsattention_maskN [UNK][PAD][MASK][MASK2]replaceFc st|trt|dddn|}t|tr4t|dddn|}t|trPt|dddn|}t|trlt|dddn|}t|trt|dddn|}t|trt|dddn|}t|trt|dddn|}t|dd}t||_W5QRXdd|jD|_| |_ t |_ dd|j D|_ t|dd}| d d d }W5QRXd d |D}tt|tt||_i|_||_td|_t|trt|dddn|}t|trt|dddn|}|dg|d<|d||g7<t|dd}t||_W5QRX| | | | fD](}||jkrtd|d|dq|j| |_|j| |_|j| |_|j| |_||_ |dks|dkr||_!n4|dkrd |_!n"|dkrd|_!ntd|d||_"t#j$f| |||||||||dddd| | | | d|dS)NF)lstriprstripTutf-8encodingcSsi|]\}}||qSr$r$r&kvr$r$r( Ksz*LukeTokenizer.__init__..cSsi|]\}}||qSr$r$rPr$r$r(rSNs rcSsg|]}t|qSr$)tuplesplit)r&merger$r$r(r)Qsz*LukeTokenizer.__init__..zJ's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+Zadditional_special_tokensz!Specified entity special token ``zT`` is not found in entity_vocab. Probably an incorrect entity vocab file is loaded: .entity_span_classificationentity_classificationentity_pair_classificationTask z| not supported. Select task from ['entity_classification', 'entity_pair_classification', 'entity_span_classification'] only.r=r>r?r@)errors bos_token eos_token unk_token sep_token cls_token pad_token mask_tokenadd_prefix_spacetaskmax_entity_lengthmax_mention_lengthentity_token_1entity_token_2entity_unk_tokenentity_pad_tokenentity_mask_tokenentity_mask2_token)% isinstancestrr openjsonloadencoderitemsdecoderr_r3 byte_encoder byte_decoderreadrWr.r/r+len bpe_rankscachergrecompilepatget entity_vocab ValueErrorentity_unk_token_identity_pad_token_identity_mask_token_identity_mask2_token_idrhrirjsuper__init__)selfrrrrhrirjrkrlrmrnrorpr_r`rarcrdrbrerfrgkwargsZ vocab_handleZ merges_handleZ bpe_mergesZentity_vocab_handleZentity_special_token __class__r$r(r&s          zLukeTokenizer.__init__cCs t|jSN)r|rvrr$r$r( vocab_sizeszLukeTokenizer.vocab_sizecCst|j}||j|Sr)r.rvcopyupdateZadded_tokens_encoder)rZvocabr$r$r( get_vocabs zLukeTokenizer.get_vocabc sd|jkrj|St|}t|}|s,|St|fddd}|jkrNqL|\}}g}d}|t|kr"z|||} Wn,tk r|||dYq"YnX|||| | }|||kr |t|dkr ||d|kr | |||d7}q^| |||d7}q^t|}|}t|dkrBqLq,t|}q,d |}|j|<|S)Ncsj|tdS)Ninf)r}rfloatpairrr$r(z#LukeTokenizer.bpe..keyrrr] ) r~rVr9minr}r|indexrextendr-join) rtokenr6r7ZbigramfirstsecondZnew_wordijr$rr(bpesB    2     zLukeTokenizer.bpecsZg}tj|D]B}dfdd|dD}|dd|dDq|S)zTokenize a string.c3s|]}j|VqdSr)ry)r&r2rr$r( sz*LukeTokenizer._tokenize..rMcss|] }|VqdSrr$)r&Z bpe_tokenr$r$r(rsr)rfindallrrencoderrrW)rtext bpe_tokensrr$rr( _tokenizes"zLukeTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rvrrb)rrr$r$r(_convert_token_to_idsz"LukeTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)rxr)rrr$r$r(_convert_id_to_tokensz"LukeTokenizer._convert_id_to_tokencs0d|}tfdd|Djdjd}|S)z:Converts a sequence of tokens (string) in a single string.rcsg|]}j|qSr$)rz)r&crr$r(r)sz:LukeTokenizer.convert_tokens_to_string..rM)r_)r bytearraydecoder_)rtokensrr$rr(convert_tokens_to_strings "z&LukeTokenizer.convert_tokens_to_string) token_ids_0 token_ids_1returncCsD|dkr|jg||jgS|jg}|jg}||||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A LUKE sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) cls_token_id sep_token_id)rrrclssepr$r$r( build_inputs_with_special_tokenss z.LukeTokenizer.build_inputs_with_special_tokens)rralready_has_special_tokensrcsh|rtj||ddS|dkr8dgdgt|dgSdgdgt|ddgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)rget_special_tokens_maskr|)rrrrrr$r(rsz%LukeTokenizer.get_special_tokens_maskcCsP|jg}|jg}|dkr.t|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. LUKE does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)rrr|)rrrrrr$r$r($create_token_type_ids_from_sequences#s z2LukeTokenizer.create_token_type_ids_from_sequencescKs>|d|j}|s|r6t|dkr6|ds6d|}||fS)Nrgrr)poprgr|isspace)rris_split_into_wordsrrgr$r$r(prepare_for_tokenization;s z&LukeTokenizer.prepare_for_tokenizationTr)r text_pair entity_spansentity_spans_pairentities entities_pairadd_special_tokenspadding truncation max_lengthristriderpad_to_multiple_ofreturn_tensorsreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboserc Kst|t}t|ttfo0t|dkp0t|dt}|sB|sBtdt|t}t|ttfort|dkprt|dt}|dks|s|stdtt|ttf}|rH|dk rtt||n|}|dkrd}n|dk rtt||n|}|dkrd}n|dk r tt||n|}|jf|||||| | | | | |||||||||d|S|j f||||||||| | | | | |||||||||d|SdS)a Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences, depending on the task you want to prepare them for. Args: text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence must be a string. Note that this tokenizer does not support tokenization based on pretokenized strings. text_pair (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence must be a string. Note that this tokenizer does not support tokenization based on pretokenized strings. entity_spans (`List[Tuple[int, int]]`, `List[List[Tuple[int, int]]]`, *optional*): The sequence or batch of sequences of entity spans to be encoded. Each sequence consists of tuples each with two integers denoting character-based start and end positions of entities. If you specify `"entity_classification"` or `"entity_pair_classification"` as the `task` argument in the constructor, the length of each sequence must be 1 or 2, respectively. If you specify `entities`, the length of each sequence must be equal to the length of each sequence of `entities`. entity_spans_pair (`List[Tuple[int, int]]`, `List[List[Tuple[int, int]]]`, *optional*): The sequence or batch of sequences of entity spans to be encoded. Each sequence consists of tuples each with two integers denoting character-based start and end positions of entities. If you specify the `task` argument in the constructor, this argument is ignored. If you specify `entities_pair`, the length of each sequence must be equal to the length of each sequence of `entities_pair`. entities (`List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences of entities to be encoded. Each sequence consists of strings representing entities, i.e., special entities (e.g., [MASK]) or entity titles of Wikipedia (e.g., Los Angeles). This argument is ignored if you specify the `task` argument in the constructor. The length of each sequence must be equal to the length of each sequence of `entity_spans`. If you specify `entity_spans` without specifying this argument, the entity sequence or the batch of entity sequences is automatically constructed by filling it with the [MASK] entity. entities_pair (`List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences of entities to be encoded. Each sequence consists of strings representing entities, i.e., special entities (e.g., [MASK]) or entity titles of Wikipedia (e.g., Los Angeles). This argument is ignored if you specify the `task` argument in the constructor. The length of each sequence must be equal to the length of each sequence of `entity_spans_pair`. If you specify `entity_spans_pair` without specifying this argument, the entity sequence or the batch of entity sequences is automatically constructed by filling it with the [MASK] entity. max_entity_length (`int`, *optional*): The maximum length of `entity_ids`. rzItext input must be of type `str` (single example) or `List[str]` (batch).NzNtext_pair input must be of type `str` (single example) or `List[str]` (batch).)batch_text_or_text_pairs(batch_entity_spans_or_entity_spans_pairs batch_entities_or_entities_pairsrrrrrirrrrrrrrrrr)rrrrrrrrrrrirrrrrrrrrrr) rqrrr*rVr|rboolr/Zbatch_encode_plusZ encode_plus) rrrrrrrrrrrrirrrrrrrrrrrrZis_valid_single_textZis_valid_batch_textZis_valid_single_text_pairZis_valid_batch_text_pairZ is_batchedrrrr$r$r(__call__AsC ( zLukeTokenizer.__call__)rrrrrrrpadding_strategytruncation_strategyrrirrrrrrrrrrrrcKsz|r td| rtd|jf||||||d|\}}}}}}|j||||||||j| j| | | ||d||||||dS)Na return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. More information on available tokenizers at https://github.com/huggingface/transformers/pull/26747is_split_into_words is not supported in this tokenizer.rrrrrrT)pair_ids entity_idspair_entity_idsentity_token_spanspair_entity_token_spansrrrrrirrrprepend_batch_axisrrrrrr)NotImplementedError_create_input_sequenceprepare_for_modelvalue)rrrrrrrrrrrrirrrrrrrrrrrr first_ids second_idsfirst_entity_idssecond_entity_idsfirst_entity_token_spanssecond_entity_token_spansr$r$r( _encode_pluss\  zLukeTokenizer._encode_plus)rrrrrrrrirrrrrrrrrrrrc)Ksf|r td| rtdg}g}g}t|D]\}}t|ttfsP|d}}n|\}}d\}}|dk r||}|rt|dtr|d}}n|\}}d\}} |dk r||}!t|!dkrt|!dtr|!\}} n |!d}} |jf|||||| d|\}"}#}$}%}&}'||"|#f||$|%f||&|'fq,|j ||||||||| | || |||| |d}(t |(S)Nzreturn_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.rNNrr)batch_entity_ids_pairsbatch_entity_token_spans_pairsrrrrrirrrrrrrrr) r enumeraterqr*rVrrr|rr-_batch_prepare_for_modelr ))rrrrrrrrrirrrrrrrrrrrrr;rrrZtext_or_text_pairrrrrZentities_or_entities_pairsrrZ"entity_spans_or_entity_spans_pairsrrrrrr batch_outputsr$r$r(_batch_encode_plus#s     z LukeTokenizer._batch_encode_plus)rrcCst|tstdn"t|dkr6t|dts6td|dk rt|tsPtdt|dkrrt|dtsrtdt|t|krtddS)Nz&entity_spans should be given as a listrz_entity_spans should be given as a list of tuples containing the start and end character indicesz7If you specify entities, they should be given as a listzGIf you specify entities, they should be given as a list of entity nameszJIf you specify entities, entities and entity_spans must be the same length)rqr*rr|rVrr)rrrr$r$r(_check_entity_input_formats   z(LukeTokenizer._check_entity_input_format)rrrrrrrc sfddfdd}d\} } d\} } d\} }jdkr|dkrN|} nF|||||\} } |dkrjgt|} nfdd|D} |dk r|dkr|} nF|||||\} }|dkrjgt|} nfdd|D} njd krt|tr.t|d kr.t|d ts6td jg} |||\} } | d \}}| d|jd g| |d} | d|jd g| |d} ||d fg} nڈjdkrt|trt|d krt|d trt|d tstd|\}}jj g} |||\} } | \}}|jd f|jd fg}|d |d kr|d |d d f| d <|d d |d df| d <t |}n4|d d |d df| d <|d |d d f| d <|D]J\\}}}| d||g| |d} | d||g| |d} qnpjdkrzt|trRt|d krRt|d tsZtd|||\} } jgt|} ntdjd| | | | | |fS)Ncsj|f}|Sr)tokenizeZconvert_tokens_to_ids)rr)rrr$r( get_input_idssz;LukeTokenizer._create_input_sequence..get_input_idscs|dkr|dfSd}g}dgt|}tttj|}i|D]T}|}|dkrl||ddkrl|d8}||kr||||7}|}t||<qD|||d7}fdd|D}||fS)Nrrrcs g|]\}}||fqSr$r$)r&Z char_startZchar_endZchar_pos2token_posr$r(r)szfLukeTokenizer._create_input_sequence..get_input_ids_and_entity_token_spans..)r|sorted frozenset itertoolschain)rrcurr;rZsplit_char_positionsZsplit_char_positionZorig_split_char_position)rrr($get_input_ids_and_entity_token_spanss.  zRLukeTokenizer._create_input_sequence..get_input_ids_and_entity_token_spansrcsg|]}j|jqSr$rrrr&entityrr$r(r)sz8LukeTokenizer._create_input_sequence..csg|]}j|jqSr$rrrr$r(r)sr[rrzsEntity spans should be a list containing a single tuple containing the start and end character indices of an entityr]r\zEntity spans should be provided as a list of two tuples, each tuple containing the start and end character indices of an entityrZz{Entity spans should be provided as a list of tuples, each tuple containing the start and end character indices of an entityr^z not supported) rhrrr|rqr*rVrZadditional_special_tokens_idsrreversed)rrrrrrrrrrrrrrrZentity_token_startZentity_token_endZ head_spanZ tail_spanZhead_token_spanZtail_token_spanZ!token_span_with_special_token_idsZspecial_token_idr$)rrrr(rs          * "           $ *z$LukeTokenizer._create_input_sequence)batch_ids_pairsrrrrrrrirrrrrrrrrrcCsi}t|||D]\}}}|\}}|\}}|\}}|j|||||||tjj|j||| dd| |||dd|d}|D]&\}}||krg||<|||qtq|j||j|| | d}t|| d}|S)a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: batch_ids_pairs: list of tokenized input ids or input ids pairs batch_entity_ids_pairs: list of entity ids or entity ids pairs batch_entity_token_spans_pairs: list of entity spans or entity spans pairs max_entity_length: The maximum length of the entity sequence. NF)rrrrrrrrrirrrrrrrrrr)rrrr tensor_type) r/rr DO_NOT_PADrrwr-padr )rrrrrrrrrirrrrrrrrrrr;rZentity_token_span_pairsrrrrrroutputsrrr$r$r(r9sZ" z&LukeTokenizer._batch_prepare_for_model)idsrrrrrrrrrrirrrrrrrrrrrrc5 s|jf|| | | |d|\}}} }tdk }t}|rDtnd}|rX|sXtd|rv|tjkrvdk rvtd|dkrd|jk}|dkrd|jk}i}|||r|j|dnd}g}|tjkr| r|| kr|j || || d \}|r ||d <|| |d <|r<| } | }!d }"td }#nD|rJn} dgt|rndgtng}!d}"t}#| |d<|r|!|d<|r|r| |d<ndgt| |d<| s|j } |dk rd}$d}%fddt||D}&fdd|D}'|$t|&7}$|%t|t|&7}%d\}(})|dk rfddt||D}(fdd|D})|$t|(7}$|%t|t|(7}%|%dkrt|%d|tjkr|$| kr|j |&|(|$| || d \}&}(}*|'dt|&}'|)dk r|)dt|(})|r|*|d<|$| |d<|(r*|&|(n|&}+t|+|d<g},g}-g}.|'|"f|)|#ffD]\}/}0|/dk rV|/D]l\}1}2|1|07}1|2|07}2tt|1|2d|j}3|3dg|j|2|17}3|,|3|-|1|.|2d qlqV|,|d<|jdkr|-|d<|.|d<|rdgt|d|d<||d| ||tjksB|rZ|j|| | |j| |d }|rpt|d|d!<t|||d"}4|4S)#ah Prepares a sequence of input id, entity id and entity span, or a pair of sequences of inputs ids, entity ids, entity spans so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids* different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error. Args: ids (`List[int]`): Tokenized input ids of the first sequence. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. entity_ids (`List[int]`, *optional*): Entity ids of the first sequence. pair_entity_ids (`List[int]`, *optional*): Entity ids of the second sequence. entity_token_spans (`List[Tuple[int, int]]`, *optional*): Entity spans of the first sequence. pair_entity_token_spans (`List[Tuple[int, int]]`, *optional*): Entity spans of the second sequence. max_entity_length (`int`, *optional*): The maximum length of the entity sequence. )rrrrrNrzAsking to return token_type_ids while setting add_special_tokens to False results in an undefined behavior. Please set add_special_tokens to True or set return_token_type_ids to None.zNot possible to return overflowing tokens for pair of sequences with the `longest_first`. Please select another truncation strategy than `longest_first`, for instance `only_second` or `only_first`.token_type_idsr<r)rZnum_tokens_to_removerroverflowing_tokensZnum_truncated_tokensrr r;special_tokens_maskcs$g|]\}}|dtkr|qSrr|r&Zent_idspanrr$r(r)sz3LukeTokenizer.prepare_for_model..cs g|]}|dtkr|qSrrr&rrr$r(r)srcs$g|]\}}|dtkr|qSrrrrr$r(r)scs g|]}|dtkr|qSrrrrr$r(r)$szb entities are ignored because their entity spans are invalid due to the truncation of input tokensoverflowing_entitiesZnum_truncated_entitiesrrUentity_position_idsrZentity_start_positionsentity_end_positionsentity_token_type_ids)rrirrrlength)r r)"_get_padding_truncation_strategiesrr|rrZ LONGEST_FIRSTmodel_input_namesZnum_special_tokens_to_addDO_NOT_TRUNCATEZtruncate_sequencesrrrrir/loggerwarningr*r+rjr-rhZ&_eventual_warn_about_too_long_sequencerr r rr )5rrrrrrrrrrrrirrrrrrrrrrrrrrrZlen_idsZ len_pair_idsencoded_inputsZ total_lenrsequencerZentity_token_offsetZpair_entity_token_offsetZtotal_entity_lenZnum_invalid_entitiesZvalid_entity_idsZvalid_entity_token_spansZvalid_pair_entity_idsZvalid_pair_entity_token_spansrZfinal_entity_idsrrrZ token_spansoffsetstartendZ position_idsrr$)rrr(rs5       &                 zLukeTokenizer.prepare_for_model) r$rrrirrrrrc  stttfr6tdtr6fdddD|jdkrftd|jddt|jd} | s|rgd<S| d} t| ttfrd} t| | dkr| d7} q| t| kr| | d} t| tttfs|t | r|dkrd n|}nZt | r |dkrd n|}n= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. rcs i|]fddDqS)csg|] }|qSr$r$)r&Zexamplerr$r(r)sz0LukeTokenizer.pad...r$)r&)r$rr(rSsz%LukeTokenizer.pad..zRYou should supply an encoding or a list of encodings to this method that includes z, but you provided r<rNtfptnpztype of z unknown: zA. Should be one of a python, numpy, pytorch or tensorflow object.)rrr)rrirrrr c3s|]}t|kVqdSrr)r&rR) batch_sizer$r(rsz$LukeTokenizer.pad..zLSome items in the output dictionary have a different batch size than others.css|]}t|VqdSrrr&inputsr$r$r(rsrcss|]}t|VqdSrrr-r$r$r(rscsi|]\}}||qSr$r$rP)rr$r(rSs)rqr*rVrkeysr rr|intrrr+Zndarraytyperwrrri_padr anyvaluesrLONGESTmaxZ MAX_LENGTHr+r-)rr$rrrirrrrZrequired_inputZ first_elementrrrr_rr.r r$)r,r$rr(r qs@        "   zLukeTokenizer.pad)r$rrirrrrc Cstd|k}|dkrd|jk}|tjkrDt|d}|rDt|d}|dk rp|dk rp||dkrp||d|}|r|dk r|dk r||dkr||d|}|tjkot|d|kp|ot|d|k}|rd|krdgt|d|d<|r|rd|krdgt|d|d<|r|t|d} |rH|t|d} |jdkrx|r|ddg| |d<|r|ddg| |d<d |kr|d dg| |d <|r|d dg| |d <d |kr|d dg| |d <|d|jg| |d<|r|d|jg| |d<|d d g|j g| |d <|j dkr|ddg| |d<|ddg| |d<n@|jdkr|rdg| |d|d<|rdg| |d|d<d |krdg| |d |d <|rdg| |d |d <d |krdg| |d |d <|jg| |d|d<|r|jg| |d|d<d g|j g| |d |d <|j dkrdg| |d|d<dg| |d|d<nt dt |j|S)a Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. max_entity_length: The maximum length of the entity sequence. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) rNr<r;rrZentity_attention_maskrightrrrrrUrZrrleftzInvalid padding strategy:) rr rr5r|r Z padding_sideZ pad_token_idrrjrhrrr) rr$rrirrrZentities_providedZneeds_to_be_padded differenceZentity_differencer$r$r(r2s$                            zLukeTokenizer._pad)save_directoryfilename_prefixrc Cstj|s"td|ddStj||r6|dndtd}tj||rX|dndtd}t|ddd $}|t j |j d d d d dW5QRXd}t|ddd j}|dt |j dddD]B\}} || krtd|d| }|d|d|d7}qW5QRXtj||r6|dndtd} t| ddd $}|t j |jd d d d dW5QRX||| fS)NzVocabulary path (z) should be a directory-rrrwrMrNr]TF)indent sort_keys ensure_asciirTrz#version: 0.2 cSs|dS)Nrr$)kvr$r$r(rrz/LukeTokenizer.save_vocabulary..rzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rrr)ospathisdirr"errorrVOCAB_FILES_NAMESrswritertdumpsrvrr}rwr#r) rr;r<rZ merge_filefrwriterrZ token_indexrr$r$r(save_vocabularys> (  (zLukeTokenizer.save_vocabulary)Nr=r>r?r@rArBrCrDrErFrGrGrFrHrIrJF)N)NF)N)F)NNNNNTFNNNrFNNNNFFFFT)NNNNN)NNNNNTFNNNrNNNNFFFFTF)TNNNNNT)N);__name__ __module__ __qualname____doc__rGZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesr rpropertyrrrrrrrrr0rrrrrrrr 'ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGrrEntitySpanInput EntityInputrrrrrr rr r!rrrrrr*rrrrrr r.r2rL __classcell__r$r$rr(r:s_t +           U l  .Q     o   $  r:)4rPrrtrCcollections.abcr functoolsrtypingrrrrrnumpyr+regexrZtokenization_utilsr Ztokenization_utils_baser r r rrrrrrrutilsrrrrZ get_loggerrMr"r0Z EntitySpanrUrrEntityrVrGrQrRrTr3r9r:r$r$r$r(sL   0   L