U ,-eE@sdZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlZddlmZmZddlmZddlmZmZmZmZmZdd lmZmZee Z!d d d d Z"ddiddiddidZ#ddiZ$GdddeZ%dS)z(Tokenization classes for OpenAI Jukebox.N)INFINITY)AnyDictListOptionalTupleUnion) AddedTokenPreTrainedTokenizer) BatchEncoding) TensorTypeis_flax_availableis_tf_availableis_torch_availablelogging)_is_jax _is_numpyz artists.jsonz lyrics.jsonz genres.json) artists_file lyrics_file genres_fileZjukeboxz=https://huggingface.co/ArthurZ/jukebox/blob/main/artists.jsonz>> from transformers import JukeboxTokenizer >>> tokenizer = JukeboxTokenizer.from_pretrained("openai/jukebox-1b-lyrics") >>> tokenizer("Alan Jackson", "Country Rock", "old town road")["input_ids"] [tensor([[ 0, 0, 0, 6785, 546, 41, 38, 30, 76, 46, 41, 49, 40, 76, 44, 41, 27, 30]]), tensor([[ 0, 0, 0, 145, 0]]), tensor([[ 0, 0, 0, 145, 0]])] ``` You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance. If nothing is provided, the genres and the artist will either be selected randomly or set to None This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to: this superclass for more information regarding those methods. However the code does not allow that and only supports composing from various genres. Args: artists_file (`str`): Path to the vocabulary file which contains a mapping between artists and ids. The default file supports both "v2" and "v3" genres_file (`str`): Path to the vocabulary file which contain a mapping between genres and ids. lyrics_file (`str`): Path to the vocabulary file which contains the accepted characters for the lyrics tokenization. version (`List[str]`, `optional`, default to `["v3", "v2", "v2"]`) : List of the tokenizer versions. The `5b-lyrics`'s top level prior model was trained using `v3` instead of `v2`. n_genres (`int`, `optional`, defaults to 1): Maximum number of genres to use for composition. max_n_lyric_tokens (`int`, `optional`, defaults to 512): Maximum number of lyric tokens to keep. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. input_idsZattention_maskv3v2rz <|endoftext|>c s0t|trt|dddn|}||_||_||_d|i|_t|dd} t | |_ W5QRXt|dd} t | |_ W5QRXt|dd} t | |_ W5QRXd} t |j dkr| dd } t| |_d d |j D|_d d |j D|_d d |j D|_tjf||||d|dS)NF)lstriprstriprutf-8encoding#[^A-Za-z0-9.,:;!?\-'\"()\[\] \t\n]+Oz\-'z\-+'cSsi|]\}}||qSr$.0kvr$r$q/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/jukebox/tokenization_jukebox.py sz-JukeboxTokenizer.__init__..cSsi|]\}}||qSr$r$r%r$r$r)r*scSsi|]\}}||qSr$r$r%r$r$r)r*s) unk_tokenn_genresversionmax_n_lyric_tokens) isinstancestrr r-r.r,Z_added_tokens_decoderopenjsonloadartists_encodergenres_encoderlyrics_encoderlenreplaceregexcompile out_of_vocabitemsartists_decodergenres_decoderlyrics_decodersuper__init__) selfrrrr-r.r,r+kwargsZ vocab_handleZoov __class__r$r)rAws4    zJukeboxTokenizer.__init__cCst|jt|jt|jS)N)r7r4r5r6rBr$r$r) vocab_sizeszJukeboxTokenizer.vocab_sizecCs|j|j|jdS)Nr4r5r6rHrFr$r$r) get_vocabszJukeboxTokenizer.get_vocabcsfdd|D}tt|D]B}fdd||D||<||dgjt||||<qfdd|dDggg}|||fS)zConverts the artist, genre and lyrics tokens to their index using the vocabulary. The total_length, offset and duration have to be provided in order to select relevant lyrics and add padding to the lyrics token sequence. csg|]}j|dqSr)r4get)r&artistrFr$r) sz9JukeboxTokenizer._convert_token_to_id..csg|]}j|dqSrJ)r5rKr&genrerFr$r)rMscsg|]}j|dqSrJ)r6rKr& characterrFr$r)rMsr)ranger7r,)rBZ list_artistsZ list_genresZ list_lyrics artists_idgenresZ lyric_idsr$rFr)_convert_token_to_ids &z%JukeboxTokenizer._convert_token_to_idcCst|S)aQ Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. Only the lyrics are split into character for the character-based vocabulary. )listrBlyricsr$r$r) _tokenizeszJukeboxTokenizer._tokenizecKs(||||\}}}||}|||fS)zV Converts three strings in a 3 sequence of tokens using the tokenizer )prepare_for_tokenizationrZ)rBrLrOrYrCr$r$r)tokenizes zJukeboxTokenizer.tokenizeF)artistsrUrYis_split_into_wordsreturncs8ttjD]l}j|dkrD||||<||g||<q||d||<fdd||dD||<qjddkrtd_d fd d ttD_ dj d <td _ j _ dd j D_ dj d<n td_|}|dd}jd|ggf}|||fS)a Performs any necessary transformations before tokenization. Args: artist (`str`): The artist name to prepare. This will mostly lower the string genres (`str`): The genre name to prepare. This will mostly lower the string. lyrics (`str`): The lyrics to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. r.v2csg|]}|dqS)r`) _normalizerNrFr$r)rMsz=JukeboxTokenizer.prepare_for_tokenization.._rrr"zOABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789.,:;!?-+'"()[] csi|]}||dqS)r$)r&index)vocabr$r)r*sz=JukeboxTokenizer.prepare_for_tokenization..zrccSsi|]\}}||qSr$r$r%r$r$r)r*sz$[^A-Za-z0-9.,:;!?\-+'\"()\[\] \t\n]+\ )rSr7r-lowerrasplitr9r:r;reZn_vocabr6r<r?_run_strip_accentsr8sub)rBr]rUrYr^idxr$)rBrer)r[s,        z)JukeboxTokenizer.prepare_for_tokenizationcCsBtd|}g}|D]"}t|}|dkr,q||qd|S)z$Strips accents from a piece of text.ZNFDZMnrf) unicodedata normalizecategoryappendjoin)rBtextoutputcharcatr$r$r)rks   z#JukeboxTokenizer._run_strip_accents)rsr_csddttdtddDddttdtddDd dttd td dDd gttd }dfdd|D}|d|d}|S)z Normalizes the input text. This process is for the genres and the artist Args: text (`str`): Artist or Genre string to normalize cSsg|] }t|qSr$chrr&ir$r$r)rMsz/JukeboxTokenizer._normalize..azrccSsg|] }t|qSr$rwryr$r$r)rMsAZcSsg|] }t|qSr$rwryr$r$r)rMs09.z_+rfcsg|]}|kr|ndqS)rbr$)r&cacceptedr$r)rMsrb) rSord frozensetrer:rrrirlstrip)rBrspatternr$rr)ras  zJukeboxTokenizer._normalize)rYr_cCs d|S)N )rrrXr$r$r)convert_lyric_tokens_to_stringsz/JukeboxTokenizer.convert_lyric_tokens_to_stringN) tensor_typeprepend_batch_axisc Cst|tst|}|tjkr@ts*tdddl}|j}|j}nh|tjkrnt sXtdddl }|j }|j}n:|tj krt stdddlm}|j}t}n tj}t}z|r|g}||s||}WntdYnX|S)a Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If unset, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. zSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rNzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.zEUnable to convert output to JAX tensors format, JAX is not installed.zUnable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.)r/r Z TENSORFLOWr ImportErrorZ tensorflowZconstant is_tensorZPYTORCHrtorchZtensorZJAXrZ jax.numpynumpyarrayrnpZasarrayr ValueError) rBinputsrrtfZ as_tensorrrZjnpr$r$r)convert_to_tensorssD       z#JukeboxTokenizer.convert_to_tensorsrfpt)r_c sdddg|gtj}|gtj}|||\}}}|||\t gtd}fddttjDt|dS)a\Convert the raw string to a list of token ids Args: artist (`str`): Name of the artist. genres (`str`): List of genres that will be mixed to condition the audio lyrics (`str`, *optional*, defaults to `""`): Lyrics used to condition the generation rrPcs6g|].}j|g||gdqS))r)rryrTZ full_tokensZ genres_idsrreturn_tensorsrBr$r)rMls z-JukeboxTokenizer.__call__..)rattention_masks)r7r-r\rVrrSr ) rBrLrUrYrZartists_tokensZ genres_tokensZ lyrics_tokensrr$rr)__call__Ys  zJukeboxTokenizer.__call__)save_directoryfilename_prefixr_c Cs"tj|s"td|ddStj||r6|dndtd}t|ddd }|t j |j d d W5QRXtj||r|dndtd }t|ddd }|t j |j d d W5QRXtj||r|dndtd }t|ddd }|t j |j d d W5QRX|||fS)a Saves the tokenizer's vocabulary dictionary to the provided save_directory. Args: save_directory (`str`): A path to the directory where to saved. It will be created if it doesn't exist. filename_prefix (`Optional[str]`, *optional*): A prefix to add to the names of the files saved by the tokenizer. zVocabulary path (z) should be a directoryN-rfrwrr F) ensure_asciirr)ospathisdirloggererrorrrVOCAB_FILES_NAMESr1writer2dumpsr4r5r6)rBrrrfrrr$r$r)save_vocabularyts,    z JukeboxTokenizer.save_vocabularycs:j|}fdd|D}fdd|D}|||fS)a Converts an index (integer) in a token (str) using the vocab. Args: artists_index (`int`): Index of the artist in its corresponding dictionary. genres_index (`Union[List[int], int]`): Index of the genre in its corresponding dictionary. lyric_index (`List[int]`): List of character indices, which each correspond to a character. csg|]}j|qSr$)r>rKrNrFr$r)rMsz9JukeboxTokenizer._convert_id_to_token..csg|]}j|qSr$)r?rKrQrFr$r)rMs)r=rK)rBZ artists_indexZ genres_indexZ lyric_indexrLrUrYr$rFr)_convert_id_to_tokens z%JukeboxTokenizer._convert_id_to_token)F)NF)rfr)N)%__name__ __module__ __qualname____doc__rZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_mapPRETRAINED_LYRIC_TOKENS_SIZESZmax_lyric_input_sizeZmodel_input_namesrApropertyrGrIrVrZr\r0boolrrrr[rkrarrrrr rr rrr __classcell__r$r$rDr)r;sH6+     -  =$r)&rr2rrrnZ json.encoderrtypingrrrrrrrrr9Ztokenization_utilsr r Ztokenization_utils_baser utilsr rrrrZ utils.genericrrZ get_loggerrrrrrrr$r$r$r)s>