U ,-e*!8@sdZddlZddlZddlmZmZddlZddlm Z ddl m Z e e Zddd Zd d id d id Zd d iZddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7d8d9d:d;dd?d@dAdBdCdDdE7ZdFdGZGdHdIdIe ZdS)Jz)Tokenization classes for Salesforce CTRL.N)OptionalTuple)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_fileZctrlzHhttps://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.jsonzHhttps://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txtiiiiii#jivi~i6iivii.iiiךiͨii%iii3iR-iniS.iKiiwii[i*iii/i?iin1iipiiiiϒi i)i-i(iiKiiiǢiihi)7Z PregnancyZ ChristianityZExplainZFitnessZSavingZAskZAssZJokeZ QuestionsZThoughtsZRetailZFeminismZWritingZAtheismZNetflixZ ComputingZOpinionZAloneFunnyZGamingZHumanZIndiaZJokerZDietZLegalZNormanZTipZWeightZMoviesZRunningZScienceZHorrorZ ConfessionZFinanceZPoliticsZScaryZSupportZ TechnologiesZTeenageEventZLearnedZNotionZ WikipediaZBooksZExtractZ ConfessionsZ ConspiracyZLinksZ NarcissusZ RelationshipZ RelationshipsZReviewsZNewsZ TranslationZ multilingualcCs>t}|d}|ddD]}|||f|}qt|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rN)setadd)wordpairsZ prev_charcharrk/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/ctrl/tokenization_ctrl.py get_pairsgsrcseZdZdZeZeZeZ e Z dfdd Z e ddZddZd d Zd d Zd dZddZddZdeeeeedddZZS) CTRLTokenizera` Construct a CTRL tokenizer. Based on Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. c st|dd}t||_W5QRXdd|jD|_t|dd}|ddd}W5QRXdd |D}tt |t t ||_ i|_ tjfd |i|dS) Nutf-8encodingcSsi|]\}}||qSrr).0kvrrr sz*CTRLTokenizer.__init__.. r cSsg|]}t|qSr)tuplesplit)rmergerrr sz*CTRLTokenizer.__init__.. unk_token)openjsonloadencoderitemsdecoderreadr!dictziprangelen bpe_rankscachesuper__init__)selfrrr$kwargsZ vocab_handleZ merges_handleZmerges __class__rrr3s zCTRLTokenizer.__init__cCs t|jSN)r/r(r4rrr vocab_sizeszCTRLTokenizer.vocab_sizecCst|jf|jSr8)r,r(Zadded_tokens_encoderr9rrr get_vocabszCTRLTokenizer.get_vocabc s|jkrj|St|}tt|dd|ddg}t|}|sN|St|fddd}|jkrpqn|\}}g}d}|t|krDz|||} Wn,tk r| ||dYqDYnX| ||| | }|||kr,|t|dkr,||d|kr,| |||d7}q| |||d7}qt|}|}t|dkrdqnqNt|}qNd |}|dd }|j|<|S) Nrzcsj|tdS)Ninf)r0getfloat)pairr9rrz#CTRLTokenizer.bpe..keyrr @@ ) r1r listrminr0r/index ValueErrorextendappendjoin) r4tokenrrZbigramfirstsecondZnew_wordijrr9rbpesF  "  2      zCTRLTokenizer.bpecCs8g}td|}|D]}|t||dq|S)zTokenize a string.z\S+\n? )refindallrKrGrSr!)r4textZ split_tokenswordsrNrrr _tokenizes  zCTRLTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)r(r=r$)r4rNrrr_convert_token_to_idsz"CTRLTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r*r=r$)r4rIrrr_convert_id_to_tokensz"CTRLTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.rTrE)rMreplacestrip)r4tokensZ out_stringrrrconvert_tokens_to_stringsz&CTRLTokenizer.convert_tokens_to_stringN)save_directoryfilename_prefixreturnc Cs(tj|s"td|ddStj||r6|dndtd}tj||rX|dndtd}t|ddd $}|t j |j d d d d dW5QRXd}t|ddd j}|dt |j dddD]B\}} || krtd|d| }|d|d|d7}qW5QRX||fS)NzVocabulary path (z) should be a directory-r\rrwrrrDTF)indent sort_keys ensure_asciirrz#version: 0.2 cSs|dS)Nr r)kvrrrr@rAz/CTRLTokenizer.save_vocabulary..rBzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rTr )ospathisdirloggererrorrMVOCAB_FILES_NAMESr%writer&dumpsr(sortedr0r)warning) r4rarbrZ merge_filefrIwriterZ bpe_tokensZ token_indexrrrsave_vocabularys2 (  zCTRLTokenizer.save_vocabulary)r)N)__name__ __module__ __qualname____doc__roZvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizes CONTROL_CODES control_codesr3propertyr:r;rSrYrZr[r`strrrrv __classcell__rrr6rrws  , r)rzr&rjtypingrrregexrUZtokenization_utilsrutilsrZ get_loggerrwrmror{r|r}rrrrrrs   ;