U -en4@sdZddlmZddlmZmZmZddlmZddl Z ddl m Z ddl m ZddlmZmZd d d d d gZeGdd d ZeGdd d Zeefeeeefeeeefeejejfeejejfeddd ZeGdd d Zeefeeeefeeeefeejejfeejejfeddd ZeeeefeejejfedddZeedddZeeeeeedddZdS)z The core algorithm(s) for processing a one or more reference and hypothesis sentences so that measures can be computed and an alignment can be visualized. ) dataclass)AnyListUnion)chainN)Opcodes) transforms) wer_default cer_defaultAlignmentChunk WordOutputCharacterOutput process_wordsprocess_charactersc@sBeZdZUdZeed<eed<eed<eed<eed<ddZd S) r a Define an alignment between two subsequence of the reference and hypothesis. Attributes: type: one of `equal`, `substitute`, `insert`, or `delete` ref_start_idx: the start index of the reference subsequence ref_end_idx: the end index of the reference subsequence hyp_start_idx: the start index of the hypothesis subsequence hyp_end_idx: the end index of the hypothesis subsequence type ref_start_idx ref_end_idx hyp_start_idx hyp_end_idxcCsn|jdkrtd|jdkr"d|_|j|jkrFtd|jd|j|j|jkrjtd|jd|jdS) N)replaceinsertdeleteequal substituterrzref_start_idx=z is larger than ref_end_idx=zhyp_start_idx=z is larger than hyp_end_idx=)r ValueErrorrrrr)selfrN/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/jiwer/process.py __post_init__Cs    zAlignmentChunk.__post_init__N)__name__ __module__ __qualname____doc__str__annotations__intrrrrrr .s  c@seZdZUdZeeeed<eeeed<eeeed<eed<eed<eed<eed<e ed <e ed <e ed <e ed <d S)r a The output of calculating the word-level levenshtein distance between one or more reference and hypothesis sentence(s). Attributes: references: The reference sentences hypotheses: The hypothesis sentences alignments: The alignment between reference and hypothesis sentences wer: The word error rate mer: The match error rate wil: The word information lost measure wip: The word information preserved measure hits: The number of correct words between reference and hypothesis sentences substitutions: The number of substitutions required to transform hypothesis sentences to reference sentences insertions: The number of insertions required to transform hypothesis sentences to reference sentences deletions: The number of deletions required to transform hypothesis sentences to reference sentences references hypotheses alignmentswermerwilwiphits substitutions insertions deletionsN r r!r"r#rr$r%r floatr&rrrrr Ys ) reference hypothesisreference_transformhypothesis_transformreturnc Cs t|tr|g}t|tr |g}tdd|Dr:tdt||dd}t||dd}t|t|krtdt|dt|d t||\}}d \}} } } d \} } g}t||D]\}}tj j ||}t d d|D}t d d|D}t dd|D}t|||}||7}| |7} | |7} | |7} | t|7} | t|7} | ddt|Dq| | | |f\}}}}t|||t|||}t|||t||||}| dkrt|| t|| nd}d|}t||||||||| | | d S)aD Compute the word-level levenshtein distance and alignment between one or more reference and hypothesis sentences. Based on the result, multiple measures can be computed, such as the word error rate. Args: reference: The reference sentence(s) hypothesis: The hypothesis sentence(s) reference_transform: The transformation(s) to apply to the reference string(s) hypothesis_transform: The transformation(s) to apply to the hypothesis string(s) Returns: (WordOutput): The processed reference and hypothesis sentences css|]}t|dkVqdS)rN)len).0trrr sz process_words..z(one or more references are empty stringsT) is_referenceFzoAfter applying the transforms on the reference and hypothesis sentences, their lengths must match. Instead got z reference and z hypothesis sentences.)rrrr)rrcss |]}|jdkrdndVqdS)rrNtagr:oprrrr<scss |]}|jdkrdndVqdS)rr>rNr?rArrrr<scss |]}|jdkrdndVqdS)rr>rNr?rArrrr<sc Ss(g|] }t|j|j|j|j|jdqS))rrrrr)r r@Z src_startZsrc_endZ dest_startZdest_endrArrr sz!process_words..r>r) r'r(r)r*r+r,r-r.r/r0r1) isinstancer$anyr_apply_transformr9 _word2charzip rapidfuzzZdistanceZ LevenshteinZeditopssumappendrZ from_editopsr3r )r4r5r6r7Zref_transformedZhyp_transformedZ ref_as_charsZ hyp_as_charsZnum_hitsZnum_substitutionsZ num_deletionsZnum_insertionsZ num_rf_wordsZ num_hp_wordsr)Zreference_sentenceZhypothesis_sentenceZedit_opsr/r1r0r.SDIHr*r+r-r,rrrrs       $c@sjeZdZUdZeeeed<eeeed<eeeed<eed<e ed<e ed<e ed<e ed <d S) r ae The output of calculating the character-level levenshtein distance between one or more reference and hypothesis sentence(s). Attributes: references: The reference sentences hypotheses: The hypothesis sentences alignments: The alignment between reference and hypothesis sentences cer: The character error rate hits: The number of correct characters between reference and hypothesis sentences substitutions: The number of substitutions required to transform hypothesis sentences to reference sentences insertions: The number of insertions required to transform hypothesis sentences to reference sentences deletions: The number of deletions required to transform hypothesis sentences to reference sentences r'r(r)cerr.r/r0r1Nr2rrrrr s c Cs6t||||}t|j|j|j|j|j|j|j|j dS)a  Compute the character-level levenshtein distance and alignment between one or more reference and hypothesis sentences. Based on the result, the character error rate can be computed. Note that the by default this method includes space (` `) as a character over which the error rate is computed. If this is not desired, the reference and hypothesis transform need to be modified. Args: reference: The reference sentence(s) hypothesis: The hypothesis sentence(s) reference_transform: The transformation(s) to apply to the reference string(s) hypothesis_transform: The transformation(s) to apply to the hypothesis string(s) Returns: (CharacterOutput): The processed reference and hypothesis sentences. )r'r(r)rPr.r/r0r1) rr r'r(r)r*r.r/r0r1)r4r5r6r7resultrrrr%s )sentence transformr=cCs:||}|r"t|dds6tdnt|dds6td|S)NT)require_non_empty_listsz~After applying the transformation, each reference should be a non-empty list of strings, with each string being a single word.FzuAfter applying the transformation, each hypothesis should be a list of strings, with each string being a single word.)_is_list_of_list_of_stringsr)rRrSr=Ztransformed_sentencerrrrFUs"rF)xrTcCsZt|tsdS|D]B}t|ts&dS|r.T)rDlistr9all)rVrTerrrrUss  rU)r4r5csbtt||}d|kr tdtt|tt|fdd|D}fdd|D}||fS)Nrz]Empty strings cannot be a word. Please ensure that the given transform removes empty strings.cs$g|]}dfdd|DqS)rcsg|]}t|qSrchrr:wZ word2charrrrCs)_word2char...joinr:rRr_rrrCsz_word2char..cs$g|]}dfdd|DqS)rcsg|]}t|qSrr[r]r_rrrCsr`rarcr_rrrCs)setrrdictrHranger9)r4r5Z vocabularyZreference_charsZhypothesis_charsrr_rrGs  rG)r# dataclassesrtypingrrr itertoolsrrIZrapidfuzz.distancerZjiwerrtrZjiwer.transformationsr r __all__r r r$ZComposeZAbstractTransformrr rboolrFrUrGrrrrsV     *. z( 1