U ,-ek9@sddlZddlmZddlmZmZmZmZmZddl Z ddl m Z m Z mZddlmZeeZdZdd iZdd d d iZd d d Zddddddddddddddddddd d!d"d#d$d%d&gZGd'd(d(eZdS))N)copyfile)AnyDictListOptionalTuple) AddedToken BatchEncodingPreTrainedTokenizer)loggingu▁ vocab_filezsentencepiece.bpe.modelzVhttps://huggingface.co/facebook/mbart-large-en-ro/resolve/main/sentencepiece.bpe.modelzUhttps://huggingface.co/facebook/mbart-large-cc25/resolve/main/sentencepiece.bpe.model)zfacebook/mbart-large-en-rozfacebook/mbart-large-cc25iZar_ARcs_CZde_DEen_XXZes_XXet_EEfi_FIZfr_XXgu_INhi_INit_ITZja_XXkk_KZko_KRlt_LTlv_LVZmy_MMne_NPZnl_XXro_ROru_RUsi_LKtr_TRvi_VNzh_CNc seZdZUdZeZeZeZ ddgZ gZ e e ed<gZe e ed<dBeeeefd fd d ZddZddZeddZeedddZejed dddZdCe e ee e ee e dfdd ZdDe e ee e e e ddd ZdEe e ee e e e dd!d"Zeeeeed#d$d%Zd&d'Z ee ed(d)d*Z!d+d,Z"d-d.Z#d/d0Z$dFeeee%ed1d2d3Z&dGe eeee eee'd6fd7d8 Z(d9d:Z)d;d<Z*d dd=d>Z+ed d?d@dAZ,Z-S)HMBartTokenizeruT Construct an MBART tokenizer. Adapted from [`RobertaTokenizer`] and [`XLNetTokenizer`]. Based on [SentencePiece](https://github.com/google/sentencepiece). The tokenization method is ` ` for source language documents, and ` ` for target language documents. Examples: ```python >>> from transformers import MBartTokenizer >>> tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-en-ro", src_lang="en_XX", tgt_lang="ro_RO") >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> inputs = tokenizer(example_english_phrase, text_target=expected_translation_romanian, return_tensors="pt") ```Z input_idsZattention_mask prefix_tokens suffix_tokensN)sp_model_kwargsc s~t|trt|dddn|}| dkr(in| _tjfj_jt||_ddddd_ d_ t j_ fd d t tD_d d jD_t jt jj j d <j jd d j D_tj| dk rfdd| Dtjf|||||||d| | jd || dk rT| nd_jj_| _jdS)NTF)lstriprstriprr)r$r'r%r&cs"i|]\}}|j|jqS) sp_model_sizefairseq_offset).0icodeselfr.m/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/mbart/tokenization_mbart.py ysz+MBartTokenizer.__init__..cSsi|]\}}||qSr.r.r1kvr.r.r6r7|sr(cSsi|]\}}||qSr.r.r8r.r.r6r7scsg|]}|kr|qSr.r.)r1t)_additional_special_tokensr.r6 sz+MBartTokenizer.__init__..) bos_token eos_token unk_token sep_token cls_token pad_token mask_tokentokenizer_filesrc_langtgt_langadditional_special_tokensr)r) isinstancestrr r)spmSentencePieceProcessorsp_modelLoadr fairseq_tokens_to_idsr0lenr/ enumerateFAIRSEQ_LANGUAGE_CODESlang_code_to_iditemsZid_to_lang_codeupdatefairseq_ids_to_tokenslistkeysextendsuper__init__ _src_langZcur_lang_code_idrGset_src_lang_special_tokens)r5r r>r?rArBr@rCrDrErFrGr)rHkwargs __class__)r<r5r6r[RsP      zMBartTokenizer.__init__cCs$|j}d|d<|j|d<|S)NrMsp_model_proto)__dict__copyrMserialized_model_proto)r5stater.r.r6 __getstate__s zMBartTokenizer.__getstate__cCs8||_t|dsi|_tjf|j|_|j|jdS)Nr))rbhasattrr)rKrLrMZLoadFromSerializedProtora)r5dr.r.r6 __setstate__s  zMBartTokenizer.__setstate__cCst|jt|j|jdS)Nr,)rPrMrSr0r4r.r.r6 vocab_sizeszMBartTokenizer.vocab_size)returncCs|jSN)r\r4r.r.r6rFszMBartTokenizer.src_lang) new_src_langrkcCs||_||jdSrl)r\r])r5rmr.r.r6rFsF) token_ids_0 token_ids_1already_has_special_tokensrkcsx|rtj||ddSdgt|j}dgt|j}|dkrT|dgt||S|dgt|dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rnrorpr,Nr)rZget_special_tokens_maskrPr"r#)r5rnrorpZ prefix_onesZ suffix_onesr_r.r6rqsz&MBartTokenizer.get_special_tokens_mask)rnrorkcCs,|dkr|j||jS|j|||jS)ab Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An MBART sequence has the following format, where `X` represents the sequence: - `input_ids` (for encoder) `X [eos, src_lang_code]` - `decoder_input_ids`: (for decoder) `X [eos, tgt_lang_code]` BOS is never used. Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N)r"r#)r5rnror.r.r6 build_inputs_with_special_tokenssz/MBartTokenizer.build_inputs_with_special_tokenscCsP|jg}|jg}|dkr.t|||dgSt||||||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. mBART does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. Nr)Z sep_token_idZ cls_token_idrP)r5rnrosepclsr.r.r6$create_token_type_ids_from_sequencess z3MBartTokenizer.create_token_type_ids_from_sequences)return_tensorsrFrGcKsJ|dks|dkrtd||_||fd|d|}||}||d<|S)zIUsed by translation pipeline, to prepare inputs for the generate functionNzATranslation requires a `src_lang` and a `tgt_lang` for this modelT)Zadd_special_tokensrvZforced_bos_token_id) ValueErrorrFZconvert_tokens_to_ids)r5Z raw_inputsrvrFrG extra_kwargsinputsZ tgt_lang_idr.r.r6_build_translation_inputs s z(MBartTokenizer._build_translation_inputscs(fddtjD}|j|S)Ncsi|]}||qSr.)Zconvert_ids_to_tokens)r1r2r4r.r6r7sz,MBartTokenizer.get_vocab..)rangerjrUZadded_tokens_encoder)r5Zvocabr.r4r6 get_vocabs zMBartTokenizer.get_vocab)textrkcCs|jj|tdS)N)Zout_type)rMencoderJ)r5r}r.r.r6 _tokenizeszMBartTokenizer._tokenizecCs4||jkr|j|S|j|}|r.||jS|jS)z0Converts a token (str) in an id using the vocab.)rOrMZ PieceToIdr0Z unk_token_id)r5tokenZspm_idr.r.r6_convert_token_to_id!s   z#MBartTokenizer._convert_token_to_idcCs&||jkr|j|S|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rVrMZ IdToPiecer0)r5indexr.r.r6_convert_id_to_token*s  z#MBartTokenizer._convert_id_to_tokencCsd|td}|S)zIConverts a sequence of tokens (strings for sub-words) in a single string. )joinreplaceSPIECE_UNDERLINEstrip)r5tokensZ out_stringr.r.r6convert_tokens_to_string0sz'MBartTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrkc Cstj|s"td|ddStj||r6|dndtd}tj|jtj|krztj |jrzt |j|n8tj |jst |d}|j }||W5QRX|fS)NzVocabulary path (z) should be a directory-rr wb)ospathisdirloggererrorrVOCAB_FILES_NAMESabspathr isfileropenrMrdwrite)r5rrZout_vocab_filefiZcontent_spiece_modelr.r.r6save_vocabulary5s (  zMBartTokenizer.save_vocabularyrr) src_textsrF tgt_textsrGrkc s||_||_tj||f|Srl)rFrGrZprepare_seq2seq_batch)r5rrFrrGr^r_r.r6rFsz$MBartTokenizer.prepare_seq2seq_batchcCs ||jSrl)r]rFr4r.r.r6_switch_to_input_modeRsz$MBartTokenizer._switch_to_input_modecCs ||jSrl)set_tgt_lang_special_tokensrGr4r.r.r6_switch_to_target_modeUsz%MBartTokenizer._switch_to_target_modecCs$|j||_g|_|j|jg|_dS)z_Reset the special tokens to the source lang setting. No prefix and suffix=[eos, src_lang_code].NrSZ cur_lang_coder"Z eos_token_idr#)r5rFr.r.r6r]Xs z*MBartTokenizer.set_src_lang_special_tokens)langrkcCs$|j||_g|_|j|jg|_dS)zcReset the special tokens to the target language setting. No prefix and suffix=[eos, tgt_lang_code].Nr)r5rr.r.r6r^s z*MBartTokenizer.set_tgt_lang_special_tokens) r$r%r%r$r&r'r(NNNNN)NF)N)N)N)rNr).__name__ __module__ __qualname____doc__rZvocab_files_names&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapZmodel_input_namesr"rint__annotations__r#rrrJrr[rfripropertyrjrFsetterboolrqrrrurzr|rrrrrrr rrrr]r __classcell__r.r.r_r6r!5s  L           r!)rshutilrtypingrrrrrZ sentencepiecerKZtokenization_utilsr r r utilsr Z get_loggerrrrrrrrRr!r.r.r.r6s$    6