U ,-e$@sdZddlZddlZddlZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZerfddlZeeZddiZdd d iiZd d iZd d ZGddde ZdS)zTokenization class for VITS.N)AnyDictListOptionalTupleUnion)PreTrainedTokenizer)is_phonemizer_availablelogging vocab_filez vocab.jsonzfacebook/mms-tts-engzChttps://huggingface.co/facebook/mms-tts-eng/resolve/main/vocab.jsonicCs td}||}|dk }|S)Nz [^\x00-\x7F])recompilesearch) input_stringZnon_roman_patternmatchZ has_non_romanrk/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/vits/tokenization_vits.pyhas_non_roman_characters/s  rc seZdZdZeZeZeZ ddgZ d$dd fd d Z e d d Z ddZddZddZd%eeeeeeeeeffdddZeeedddZeeedddZddZdd Zd&eeeeeedfd!d"d#ZZS)' VitsTokenizera Construct a VITS tokenizer. Also supports MMS-TTS. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. language (`str`, *optional*): Language identifier. add_blank (`bool`, *optional*, defaults to `True`): Whether to insert token id 0 in between the other tokens. normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the input text by removing all casing and punctuation. phonemize (`bool`, *optional*, defaults to `True`): Whether to convert the input text into phonemes. is_uroman (`bool`, *optional*, defaults to `False`): Whether the `uroman` Romanizer needs to be applied to the input text prior to tokenizing. Z input_idsZattention_maskNTF)returnc s~t|dd} t| |_W5QRXdd|jD|_||_||_||_||_ ||_ t j f|||||||d| dS)Nutf-8encodingcSsi|]\}}||qSrr).0kvrrr csz*VitsTokenizer.__init__..) pad_token unk_tokenlanguage add_blank normalize phonemize is_uroman) openjsonloadencoderitemsdecoderr"r#r$r%r&super__init__) selfr r r!r"r#r$r%r&kwargsZ vocab_handle __class__rrr.Ts& zVitsTokenizer.__init__cCs t|jSN)lenr*r/rrr vocab_sizevszVitsTokenizer.vocab_sizecs(fddtjD}|j|S)Ncsi|]}||qSr)Zconvert_ids_to_tokens)rir5rrr{sz+VitsTokenizer.get_vocab..)ranger6updateadded_tokens_encoder)r/Zvocabrr5r get_vocabzs zVitsTokenizer.get_vocabcCst|jt|j}d}d}|t|krd}|D]8}|||t||kr8||7}|t|7}d}qrq8|s$|||7}|d7}q$|S)zfLowercase the input string, respecting any special token ids that may be part or entirely upper-cased.rFT)listr*keysr:r4lower)r/rZall_vocabulary filtered_textr7 found_matchwordrrrnormalize_texts   zVitsTokenizer.normalize_textcCs|jdkr|dd}|S)z4Special treatment of characters in certain languagesZronuțuţ)r"replace)r/textrrr_preprocess_chars  zVitsTokenizer._preprocess_char)rFis_split_into_wordsr$rc s|dk r |nj}|r |}|}t|rBjrBtdjr|tsVt dt j|dddddd}t dd |}n$|rd ttfd d |}||fS) a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. normalize (`bool`, *optional*, defaults to `None`): Whether or not to apply punctuation and casing normalization to the text inputs. Typically, VITS is trained on lower-cased and un-punctuated text. Hence, normalization is used to ensure that the input text consists only of lower-case characters. kwargs (`Dict[str, Any]`, *optional*): Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. NzText to the tokenizer contains non-Roman characters. Ensure the `uroman` Romanizer is applied to the text prior to passing it to the tokenizer. See `https://github.com/isi-nlp/uroman` for details.zEPlease install the `phonemizer` Python package to use this tokenizer.zen-usZespeakT)r"backendstripZpreserve_punctuationZ with_stressz\s+ r<cs |jkSr3)r*)charr5rrz8VitsTokenizer.prepare_for_tokenization..)r$rDrGrr&loggerwarningr%r ImportError phonemizerr subjoinr>filterrJ)r/rFrHr$r0rArr5rprepare_for_tokenizations.   z&VitsTokenizer.prepare_for_tokenization)rFrcCs@t|}|jr<|dgt|dd}||ddd<|}|S)z]Tokenize a string by inserting the `` token at the boundary between adjacent characters.rr=N)r>r#_convert_id_to_tokenr4)r/rFtokensZ interspersedrrr _tokenizes zVitsTokenizer._tokenize)rYrcCs*|jr t|dkr |ddd}d|S)Nr=rWr<)r#r4rT)r/rYrrrconvert_tokens_to_stringsz&VitsTokenizer.convert_tokens_to_stringcCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r*getr!)r/tokenrrr_convert_token_to_idsz"VitsTokenizer._convert_token_to_idcCs |j|S)z=Converts an index (integer) in a token (str) using the vocab.)r,r\)r/indexrrrrXsz"VitsTokenizer._convert_id_to_token)save_directoryfilename_prefixrc Cstj|s"td|ddStj||r6|dndtd}t|ddd$}|t j |j d d d d d W5QRX|fS)NzVocabulary path (z) should be a directory-r<r wrrrWTF)indent sort_keys ensure_ascii ) ospathisdirrOerrorrTVOCAB_FILES_NAMESr'writer(dumpsr*)r/r`rar frrrsave_vocabularys (zVitsTokenizer.save_vocabulary)rrNTTTF)FN)N) __name__ __module__ __qualname____doc__rlZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesZmodel_input_namesr.propertyr6r;rDrGstrboolrrrrrVrrZr[r^rXrrp __classcell__rrr1rr9sB "  < r)rtr(rhr typingrrrrrrZtokenization_utilsr utilsr r rRZ get_loggerrqrOrlrurvrrrrrrs(