U
,-e @ sb d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
m Z m Z m
Z
mZmZ ddlZddlmZ ddlmZmZ dd lmZmZmZmZm Z m Z m Z m Z m Z e !e"Z#e
re rddl$Z$e rddl%Z&e rddl'mZ( d
d d Z)d
did
did Z*d
ej+iZ,dZ-e e e.ee/e.f f Z0eG dd deZ1G dd deZ2G dd deZ3dS )z Tokenization class for Wav2Vec2. N) dataclass)groupby)
TYPE_CHECKINGDictListOptionalTupleUnion )PreTrainedTokenizer)
AddedToken
BatchEncoding) ModelOutputPaddingStrategy
TensorTypeadd_end_docstringsis_flax_availableis_tf_availableis_torch_availablelogging to_py_objz
vocab.jsonztokenizer_config.json
vocab_fileZtokenizer_config_filefacebook/wav2vec2-base-960hJhttps://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/vocab.jsonzUhttps://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/tokenizer_config.jsona
padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):
Activates and controls padding. Accepts the following values:
- `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
sequence if provided).
- `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
acceptable input length for the model if that argument is not provided.
- `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different
lengths).
max_length (`int`, *optional*):
Controls the maximum length to use by one of the truncation/padding parameters.
If left unset or set to `None`, this will use the predefined model maximum length if a maximum length
is required by one of the truncation/padding parameters. If the model has no specific maximum input
length (like XLNet) truncation/padding to a maximum length will be deactivated.
pad_to_multiple_of (`int`, *optional*):
If set will pad the sequence to a multiple of the provided value. This is especially useful to enable
the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).
return_tensors (`str` or [`~utils.TensorType`], *optional*):
If set, will return tensors instead of list of python integers. Acceptable values are:
- `'tf'`: Return TensorFlow `tf.constant` objects.
- `'pt'`: Return PyTorch `torch.Tensor` objects.
- `'np'`: Return Numpy `np.ndarray` objects.
verbose (`bool`, *optional*, defaults to `True`):
Whether or not to print more information and warnings.
c @ sV e Zd ZU dZeee ef ed< dZeee e f ed< dZ
eee e f ed< dS )Wav2Vec2CTCTokenizerOutputaN
Output type of [` Wav2Vec2CTCTokenizer`], with transcription.
Args:
text (list of `str` or `str`):
Decoded logits in text from. Usually the speech transcription.
char_offsets (list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]`):
Offsets of the decoded characters. In combination with sampling rate and model downsampling rate char
offsets can be used to compute time stamps for each charater. Total logit score of the beam associated with
produced text.
word_offsets (list of `List[Dict[str, Union[int, str]]]` or `List[Dict[str, Union[int, str]]]`):
Offsets of the decoded words. In combination with sampling rate and model downsampling rate word offsets
can be used to compute time stamps for each word.
textN char_offsets word_offsets) __name__
__module__ __qualname____doc__r r str__annotations__r
ListOfDictr r& r& s/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/transformers/models/wav2vec2/tokenization_wav2vec2.pyr i s
r c sF e Zd ZdZeZeZeZ ddgZ
dB fd d
Z e dddZ
ee dddZeee dddZejdd Zejdd ZeedddZedddZdCeee ee f eed fd d
Zd d Z e ed!d"d#Z ee d$d%d&Z dDee eeeeee ee e f f d(d)d*Z e!ee ee eeee ee ef f d+d,d-Z"e!dEee ee e f f e ee ee e f f d.d/d0Z#dFd1d2Z$dGee eeeeee ee e d3d4d5Z%dHeee eee d6d7d8f eeeeee d9d:d;Z&dIeeee d6d7d8f eeeee d<d=d>Z'dJe ee e(e d?d@dAZ) Z*S )KWav2Vec2CTCTokenizera
Constructs a Wav2Vec2CTC tokenizer.
This tokenizer inherits from [`PreTrainedTokenizer`] which contains some of the main methods. Users should refer to
the superclass for more information regarding such methods.
Args:
vocab_file (`str`):
File containing the vocabulary.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sentence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sentence token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
word_delimiter_token (`str`, *optional*, defaults to `"|"`):
The token used for defining the end of a word.
do_lower_case (`bool`, *optional*, defaults to `False`):
Whether or not to accept lowercase input and lowercase the output when decoding.
target_lang (`str`, *optional*):
A target language the tokenizer should set by default. `target_lang` has to be defined for multi-lingual,
nested vocabulary such as [facebook/mms-1b-all](https://huggingface.co/facebook/mms-1b-all).
**kwargs
Additional keyword arguments passed along to [`PreTrainedTokenizer`]
Z input_idsattention_mask"`):
The beginning of sentence token.
eos_token (`str`, *optional*, defaults to `""`):
The end of sentence token.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
word_delimiter_token (`str`, *optional*, defaults to `"|"`):
The token used for defining the end of a word.
do_lower_case (`bool`, *optional*, defaults to `False`):
Whether or not to lowercase the output when decoding.
do_normalize (`bool`, *optional*, defaults to `False`):
Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly
improve the performance for some models, *e.g.*,
[wav2vec2-lv60](https://huggingface.co/models?search=lv60).
return_attention_mask (`bool`, *optional*, defaults to `False`):
Whether or not [`~Wav2Vec2Tokenizer.__call__`] should return `attention_mask`.
Wav2Vec2 models that have set `config.feat_extract_norm == "group"`, such as
[wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base-960h), have **not** been trained using
`attention_mask`. For such models, `input_values` should simply be padded with 0 and no `attention_mask`
should be passed.
For Wav2Vec2 models that have set `config.feat_extract_norm == "layer"`, such as
[wav2vec2-lv60](https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self), `attention_mask` should be
passed for batched inference.
**kwargs
Additional keyword arguments passed along to [`PreTrainedTokenizer`]
r r zNhttps://huggingface.co/facebook/wav2vec2-base-960h/resolve/main/tokenizer.jsonr input_valuesr) r* r+ r, r- r. Fc
s t dt || _|| _| | _|| _t|dd} t | | _
W 5 Q R X dd | j
D | _ t
jf ||||||| |d|
d S )NzThe class `Wav2Vec2Tokenizer` is deprecated and will be removed in version 5 of Transformers. Please use `Wav2Vec2Processor` or `Wav2Vec2CTCTokenizer` instead.r0 r1 c S s i | ]\}}||qS r& r& r3 r& r& r' r7 s z.Wav2Vec2Tokenizer.__init__..)r8 r9 r: r; r< do_normalizereturn_attention_maskr= )warningswarn
FutureWarningrD r< r r rE rF rG rI rJ rK rL rM ) rR r r9 r: r8 r; r= r< r r rS rT rV r& r' rM s.
zWav2Vec2Tokenizer.__init__r^ c C s( | j dkr | jr td dS t| j S )zW
`str`: Padding token. Log an error if used while not having been set.
Nr` ra re r& r& r' r= s
z&Wav2Vec2Tokenizer.word_delimiter_tokenc C s | j dkrdS | | jS rf rg re r& r& r' ri s
z)Wav2Vec2Tokenizer.word_delimiter_token_idc C s
|| _ d S rj rk rl r& r& r' r= s c C s | || _d S rj rn rl r& r& r' ri s NT)
raw_speechpadding
max_lengthpad_to_multiple_ofreturn_tensorsrb r_ c K s t |tjot|jdk}|r:t|jdkr:td| |p`t |ttfo`t |d tjttf} | rt |d tjsdd |D }n| st |tjst|}| s|g}| j rdd |D }t
d|i}
| j |
|||| j ||d } | S )
a
Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of
sequences.
Args:
raw_speech (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`):
The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
values, a list of numpy array or a list of list of float values. Must be mono channel audio, not
stereo, i.e. single float per timestep.
r@ r z2Only mono-channel audio is supported for input to r c S s g | ]}t |qS r& )r r )r4 Zspeechr& r& r' rZ 8 s z.Wav2Vec2Tokenizer.__call__..c S s. g | ]&}|t | t t |d qS )gh㈵>)r Zmeansqrtvar)r4 xr& r& r' rZ B s r )r r r r r rb )
rx r ndarrayrO shaper[ r{ tupler r r
padr ) rR r r r r r rb rS Zis_batched_numpyZ
is_batchedZencoded_inputsZ
padded_inputsr& r& r' __call__ s2 "
zWav2Vec2Tokenizer.__call__c C s
t | jS rj ro re r& r& r' rp S s z Wav2Vec2Tokenizer.vocab_sizec C s t | jf| jS rj )rq rI rs re r& r& r' rt W s zWav2Vec2Tokenizer.get_vocabr~ c C s | j || j | jS r r r r& r& r' r Z s z&Wav2Vec2Tokenizer._convert_token_to_idr c C s | j || j}|S r r r r& r& r' r ^ s z&Wav2Vec2Tokenizer._convert_id_to_token)r r_ c sV dd t |D }tt fdd|}d fdd|D } jrR| }|S )r c S s g | ] }|d qS )r r& )r4 Z token_groupr& r& r' rZ h s z>Wav2Vec2Tokenizer.convert_tokens_to_string..c s
| j kS rj r )rU re r& r' r k r z.r c s g | ]}| j krd n|qS )r/ )r= rY re r& r' rZ n s )r r{ r r\ r r< r )rR r Zgrouped_tokensr r r& re r' r c s z*Wav2Vec2Tokenizer.convert_tokens_to_string)r r r r_ c
K sl | j ||d}g }|D ] }|r*|| jkr*q|| q| |}|dk rL|n| j}|rd| |} | S |S dS )r r N)r r rP r r r )
rR r r r rS r r rU r Z
clean_textr& r& r' r u s
zWav2Vec2Tokenizer._decoder c C s t j|s"td| d d S t j||r6|d ndtd }t|ddd$}|t j
| j d d
d d d
W 5 Q R X |fS r ) r r r rc rd r\ r rE r rF r rI r r& r& r' r s (z!Wav2Vec2Tokenizer.save_vocabulary)r* r+ r, r- r. FFF)FNNNT)FN)N)&r r r! r" r r r r rM r r# r= r r ri r r WAV2VEC2_KWARGS_DOCSTRINGr r r r r r r r r
r rp r rt r r r r r r r r& r& rV r' r sr .
*
" 9 !r )4r" rF r sysr dataclassesr itertoolsr typingr r r r r r numpyr Ztokenization_utilsr Ztokenization_utils_baser r
utilsr r r r r r r r r Z
get_loggerr rc ZtorchZ
tensorflowtfZ jax.numpyZjnpr r maxsizer r r# r r% r r( r r& r& r& r' sP ,