U -e @sddlmZmZddlmZmZddlZddlmZ ddlm Z ddlm Z GdddeZ Gdd d e Z de eeed d d ZGd ddeZddZddZGdddeZdS))ABCabstractmethod)DictListN)Tensor) TokenSpanc@s,eZdZeeeeeedddZdS) ITokenizer transcriptreturncCsdS)a Tokenize the given transcript (list of word) .. note:: The toranscript must be normalized. Args: transcript (list of str): Transcript (list of word). Returns: (list of int): List of token sequences Nselfr r r g/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torchaudio/pipelines/_wav2vec2/aligner.py__call__ szITokenizer.__call__N)__name__ __module__ __qualname__rrstrrr r r rr src@s>eZdZeeefdddZeeeeedddZdS) Tokenizer dictionarycCs ||_dSNr)rrr r r__init__szTokenizer.__init__r csfdd|DS)Ncsg|]}fdd|DqS)csg|]}j|qSr r).0crr r sz1Tokenizer.__call__...r )rwordrr rr sz&Tokenizer.__call__..r r r rrrszTokenizer.__call__N) rrrrrintrrrr r r rrsr)emissiontokensblankcCsZ|j}|d}tj|gtj|d}tj|||d\}}|}|d|d}}||fS)Nr)Zdtypedevicer")r#Z unsqueezetorchZtensorZint32FZ forced_alignexp)r r!r"r#targetsaligned_tokensscoresr r r_align_emission_and_tokens#s r+c@s2eZdZeeeeeeeedddZdS)IAlignerr r!r cCsdS)aGenerate list of time-stamped token sequences Args: emission (Tensor): Sequence of token probability distributions in log-domain. Shape: `(time, tokens)`. tokens (list of integer sequence): Tokenized transcript. Output from :py:class:`Wav2Vec2FABundle.Tokenizer`. Returns: (list of TokenSpan sequence): Tokens with time stamps and scores. Nr )rr r!r r rr0szIAligner.__call__N) rrrrrrrrrr r r rr,/sr,cCsHt|t|kstd}g}|D]"}|||||||7}q |S)Nr)lensumAssertionErrorappend)list_lengthsiretlr r r _unflatten?s r7cCsdd|DS)NcSsg|]}|D]}|q qSr r )rr2itemr r rrJsz_flatten..r )Z nested_listr r r_flattenIsr9c@s6eZdZddZeeeeeeedddZdS)AlignercCs ||_dSrr$)rr"r r rrNszAligner.__init__r-cCsP|jdkrtd|jt|t||j\}}t||}t|dd|DS)Nz&The input emission must be 2D. Found: cSsg|] }t|qSr )r.)rtsr r rrWsz$Aligner.__call__..) ndim ValueErrorshaper+r9r"r&Z merge_tokensr7)rr r!r)r*spansr r rrQs   zAligner.__call__N) rrrrrrrrrr r r rr:Msr:)r)abcrrtypingrrr%Ztorchaudio.functionalZ functionalr&rrrrrr+r,r7r9r:r r r rs