U
,-eEk @ sR d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
Z
ddl m Z ddl
mZ eeZdd d
Zd d id d
id
Zd diZdd ZG dd de ZdZdZededdddddf Ze
d d e e
j e
j B e
j B Z e
d Z e
ee
j e
j B e
j B Z!e
d Z"d1d!d"Z#d2d&d'Z$G d(d) d)Z%d*d+ Z&d,d- Z'd3d/d0Z(dS )4z" Tokenization classes for BERTweet N)copyfile)ListOptionalTuple )PreTrainedTokenizer)loggingz vocab.txtz bpe.codes)
vocab_file merges_filezvinai/bertweet-basezAhttps://huggingface.co/vinai/bertweet-base/resolve/main/vocab.txtzAhttps://huggingface.co/vinai/bertweet-base/resolve/main/bpe.codes c C s> t }| d }| dd D ]}|||f |}qt |}|S )z
Return set of symbol pairs in a word.
Word is represented as tuple of symbols (symbols being variable-length strings).
r N)setadd)wordpairsZ prev_charchar r s/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/bertweet/tokenization_bertweet.py get_pairs4 s r c s e Zd ZdZeZeZeZ d* fdd Z
d+e e e
e e e e d d d
Zd,e e e
e e ee e d fdd
Zd-e e e
e e e e d ddZedd Zdd Zdd Zdd Zdd Zd d Zd d Zd!d" Zd#d$ Zd.e e
e e e d%d&d'Z d(d) Z Z S )/BertweetTokenizera
Constructs a BERTweet tokenizer, using Byte-Pair-Encoding.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
merges_file (`str`):
Path to the merges file.
normalization (`bool`, *optional*, defaults to `False`)
Whether or not to apply a normalization preprocess.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
mask_token (`str`, *optional*, defaults to `""`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
F X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
N) cls_token_id sep_token_id)rI rM rN clssepr r r build_inputs_with_special_tokens s
z2BertweetTokenizer.build_inputs_with_special_tokens)rM rN already_has_special_tokensrO c sh |rt j||ddS |dkr8dgdgt| dg S dgdgt| ddg dgt| dg S )a
Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding
special tokens using the tokenizer `prepare_for_model` method.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
already_has_special_tokens (`bool`, *optional*, defaults to `False`):
Whether or not the token list is already formatted with special tokens for the model.
Returns:
`List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
T)rM rN rU Nr r )rG get_special_tokens_maskrA )rI rM rN rU rK r r rV s z)BertweetTokenizer.get_special_tokens_maskc C sP | j g}| jg}|dkr.t|| | dg S t|| | | | | dg S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. BERTweet does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`List[int]`):
List of IDs.
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: List of zeros.
Nr )rQ rP rA )rI rM rN rS rR r r r $create_token_type_ids_from_sequences s
z6BertweetTokenizer.create_token_type_ids_from_sequencesc C s
t | jS N)rA r8 rI r r r
vocab_size s z BertweetTokenizer.vocab_sizec C s t | jf| jS rX )r> r8 Zadded_tokens_encoderrY r r r get_vocab s zBertweetTokenizer.get_vocabc
s | j kr j | S t|}tt|d d |d d g }t|}|sN|S t| fddd}| jkrpqn|\}}g }d}|t|k rDz|||} W n, tk
r | ||d Y qDY nX | ||| | }|| |kr,|t|d k r,||d |kr,|
|| |d7 }q|
|| |d7 }qt|}|}t|dkrdqnqNt|}qNd |}|d d
}| j |< |S ) Nr% zc s j | tdS )Ninf)rB getfloat)pairrY r r z'BertweetTokenizer.bpe..)keyr r r @@ ) rC r& listr minrB rA index
ValueErrorextendappendjoin)
rI tokenr r ZbigramfirstsecondZnew_wordijr rY r bpe sF
"
2
zBertweetTokenizer.bpec C sH | j r| |}g }td|}|D ] }|t| |d q$|S )zTokenize a string.z\S+\n? )r+ normalizeTweetrefindallri re rq r' )rI textZ split_tokenswordsrl r r r _tokenize8 s
zBertweetTokenizer._tokenizec s j D ]}|| j | }q j|}d fdd|D }|ddddddd d
d d }|d
ddddddddddd}|dddd d d d d }d| S )!z'
Normalize a raw Tweet
rr c s g | ]} |qS r )normalizeToken)r rl rY r r r) K s z4BertweetTokenizer.normalizeTweet..zcannot zcan not zn't z n't zn 't zca n'tzcan'tzai n'tzain'tz'm z 'm z're z 're z's z 's z'll z 'll z'd z 'd z've z 've z p . m .z p.m.z p . m z p.m z a . m .z a.m.z a . m z a.m )rF replacerE tokenizerk r' )rI ZtweetpuncttokensZ normTweetr rY r rs C s^
z BertweetTokenizer.normalizeTweetc C sp | }|drdS |ds*|dr.dS t|dkrh|| jkrN| j| S | jdk rb| |S |S n|S dS )z-
Normalize tokens in a Tweet
@z@USERhttpZwwwZHTTPURLr N)lower
startswithrA rF r4 )rI rl Zlowercased_tokenr r r ry e s
z BertweetTokenizer.normalizeTokenc C s | j || j | jS )z0Converts a token (str) in an id using the vocab.)r8 r] r0 )rI rl r r r _convert_token_to_idx s z&BertweetTokenizer._convert_token_to_idc C s | j || jS )z=Converts an index (integer) in a token (str) using the vocab.)r; r] r0 )rI rg r r r _convert_id_to_token| s z&BertweetTokenizer._convert_id_to_tokenc C s d |dd }|S )z:Converts a sequence of tokens (string) in a single string.rr rc )rk rz strip)rI r} Z
out_stringr r r convert_tokens_to_string s z*BertweetTokenizer.convert_tokens_to_string)save_directoryfilename_prefixrO c C s t j|s"td| d d S t j||r6|d ndtd }t j||rX|d ndtd }t j| jt j|krt j | jrt
| j| n8t j | jst |d}| j
}|| W 5 Q R X t j| jt j|krt
| j| ||fS )NzVocabulary path (z) should be a directory-r r r
wb)ospathisdirr6 errorrk VOCAB_FILES_NAMESabspathr isfiler r< Zsp_modelZserialized_model_protowriter
)rI r r Zout_vocab_fileZout_merge_filefiZcontent_spiece_modelr r r save_vocabulary s( (
z!BertweetTokenizer.save_vocabularyc
C s t |trz(t|ddd}| | W 5 Q R X W nH tk
rX } z|W 5 d}~X Y n$ tk
rz td| dY nX dS | }|D ]B}| }| d}|dkrt
d |d| }t | j | j |< qdS )
zi
Loads a pre-existing dictionary from a text file and adds its symbols to this instance.
rr! r" Nz Incorrect encoding detected in z , please rebuild the datasetrr r% z5Incorrect dictionary format, expected ' ')
isinstancestrr< r9 FileNotFoundError UnicodeError Exception readlinesr rfindrh rA r8 ) rI ffdZfnfelinesZlineTmplineidxr r r r r9 s"
z BertweetTokenizer.add_from_file)Fr r r r r r r )N)NF)N)N)!__name__
__module__ __qualname____doc__r Zvocab_files_namesPRETRAINED_VOCAB_FILES_MAPZpretrained_vocab_files_map&PRETRAINED_POSITIONAL_EMBEDDINGS_SIZESZmax_model_input_sizesrH r intr rT boolrV rW propertyrZ r[ rq rx rs ry r r r r r r r9
__classcell__r r rK r r D sZ 2 =
, " r ac
(?:
[<>]?
[:;=8] # eyes
[\-o\*\']? # optional nose
[\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
|
[\)\]\(\[dDpP/\:\}\{@\|\\] # mouth
[\-o\*\']? # optional nose
[:;=8] # eyes
[<>]?
|
<3 # heart
)u # Capture 1: entire matched URL
(?:
https?: # URL protocol and colon
(?:
/{1,3} # 1-3 slashes
| # or
[a-z0-9%] # Single letter or digit or '%'
# (Trying not to match e.g. "URI::Escape")
)
| # or
# looks like domain name followed by a slash:
[a-z0-9.\-]+[.]
(?:[a-z]{2,13})
/
)
(?: # One or more:
[^\s()<>{}\[\]]+ # Run of non-space, non-()<>{}[]
| # or
\([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...)
|
\([^\s]+?\) # balanced parens, non-recursive: (...)
)+
(?: # End with:
\([^\s()]*?\([^\s()]+\)[^\s()]*?\) # balanced parens, one level deep: (...(...)...)
|
\([^\s]+?\) # balanced parens, non-recursive: (...)
| # or
[^\s`!()\[\]{};:'".,<>?«»“”‘’] # not a space or one of these punct chars
)
| # OR, the following to match naked domains:
(?:
(?\s]+>z
[\-]+>|<[\-]+z (?:@[\w_]+)z (?:\#+[\w_]+[\w\'_\-]*[\w_]+)z#[\w.+-]+@[\w-]+\.(?:[\w-]\.?)+[\w-]a
(?:[^\W\d_](?:[^\W\d_]|['\-_])+[^\W\d_]) # Words with apostrophes or dashes.
|
(?:[+\-]?\d+[,/.:-]\d+[+\-]?) # Numbers, including fractions, decimals.
|
(?:[\w_]+) # Words without apostrophes or dashes.
|
(?:\.(?:\s*\.){1,}) # Ellipsis dots.
|
(?:\S) # Everything else that isn't whitespace.
z(%s)|z([^a-zA-Z0-9])\1{3,}z&(#?(x?))([^&;\s]+);strictc C s& |d kr d}t | tr"| ||S | S )Nr! )r bytesdecode)rv r# errorsr r r _str_to_unicodel s
r r Tr! c s fdd}t |t| |S )u
Remove entities from text by converting them to their corresponding unicode character.
Args:
text:
A unicode string or a byte string encoded in the given *encoding* (which defaults to 'utf-8').
keep (list):
List of entity names which should not be replaced. This supports both numeric entities (`&#nnnn;` and
`&#hhhh;`) and named entities (such as ` ` or `>`).
remove_illegal (bool):
If `True`, entities that can't be converted are removed. Otherwise, entities that can't be converted are
kept "as is".
Returns: A unicode string with the entities removed.
See https://github.com/scrapy/w3lib/blob/master/w3lib/html.py
Examples:
```python
>>> from nltk.tokenize.casual import _replace_html_entities
>>> _replace_html_entities(b"Price: £100")
'Price: \xa3100'
>>> print(_replace_html_entities(b"Price: £100"))
Price: £100
```c s | d}| dr~zN| dr,t|d}n
t|d}d| krJdkr`n nt|fdW S W q tk
rz d }Y qX n | kr| d S tjj|}|d k rz
t |W S tt
fk
r Y nX rd
S | d S ) Nr r r
r cp1252r r ) groupr r r rh htmlentitiesname2codepointr] chr
OverflowError)matchZ entity_bodynumberkeepremove_illegalr r _convert_entity s&
z/_replace_html_entities.._convert_entity)ENT_REsubr )rv r r r# r r r r _replace_html_entitiest s r c @ s" e Zd ZdZd ddZdd ZdS )
rD a
Examples:
```python
>>> # Tokenizer for tweets.
>>> from nltk.tokenize import TweetTokenizer
>>> tknzr = TweetTokenizer()
>>> s0 = "This is a cooool #dummysmiley: :-) :-P <3 and some arrows < > -> <--"
>>> tknzr.tokenize(s0)
['This', 'is', 'a', 'cooool', '#dummysmiley', ':', ':-)', ':-P', '<3', 'and', 'some', 'arrows', '<', '>', '->', '<--']
>>> # Examples using *strip_handles* and *reduce_len parameters*:
>>> tknzr = TweetTokenizer(strip_handles=True, reduce_len=True)
>>> s1 = "@remy: This is waaaaayyyy too much for you!!!!!!"
>>> tknzr.tokenize(s1)
[':', 'This', 'is', 'waaayyy', 'too', 'much', 'for', 'you', '!', '!', '!']
```TFc C s || _ || _|| _d S rX
preserve_case
reduce_len
strip_handles)rI r r r r r r rH s zTweetTokenizer.__init__c C sR t |}| jrt|}| jr$t|}td|}t|}| j sNdd |D }|S )z
Args:
text: str
Returns: list(str) A tokenized list of strings; concatenating this list returns the original string if
`preserve_case=False`
\1\1\1c S s" g | ]}t |r|n| qS r ) EMOTICON_REsearchr )r xr r r r) s z+TweetTokenizer.tokenize..)
r r remove_handlesr reduce_lengtheningHANG_REr WORD_REru r )rI rv Z safe_textrw r r r r{ s
zTweetTokenizer.tokenizeN)TFF)r r r r rH r{ r r r r rD s
rD c C s t d}|d| S )za
Replace repeated character sequences of length 3 or greater with sequences of length 3.
z (.)\1{2,}r regexcompiler rv patternr r r r s
r c C s t d}|d| S )z4
Remove Twitter username handles from text.
zv(? sb
*.0$
A8