U -eE@sdZddlZddlZddlZddlZddlZddlmZmZm Z dddddd d d d d dddddddgZ Gddde Z Gddde Z Gddde ZGddde ZGdd d e ZGdd d e ZGddde ZGddde ZGdddeZGdddeZed d!d"ZGd#d d eZGd$d d e ZGd%dde ZGd&dde ZGd'dde ZGd(dde ZGd)dde ZGd*d d e ZdS)+a@ This file implements the building blocks for transforming a collection of input strings to the desired format in order to calculate the WER of CER. In principle, for word error rate calculations, every string of a sentence needs to be collapsed into a list of strings, where each string is a *single* word. This is done with [transforms.ReduceToListOfListOfWords][]. A composition of multiple transformations must therefore *always* end with [transforms.ReduceToListOfListOfWords][]. For the character error rate, every string of a sentence also needs to be collapsed into a list of strings, but here each string is a *single* character. This is done with [transforms.ReduceToListOfListOfChars][]. Similarly, a composition of multiple transformations must therefore also always end with [transforms.ReduceToListOfListOfChars][]. N)UnionListMappingAbstractTransformComposeExpandCommonEnglishContractionsRemoveEmptyStringsReduceToListOfListOfWordsReduceToListOfListOfCharsReduceToSingleSentenceRemoveKaldiNonWordsRemoveMultipleSpacesRemovePunctuationRemoveSpecificWordsRemoveWhiteSpaceStripSubstituteRegexesSubstituteWords ToLowerCase ToUpperCasec@sJeZdZdZeeeefdddZedddZeedd d Z d S) rz( The base class of a Transform. ) sentencescCs:t|tr||St|tr(||Std|dS)z Transforms one or more strings. Args: sentences: The strings to transform. Returns: (Union[str, List[str]]): The transformed strings. z7input {} was expected to be a string or list of stringsN) isinstancestrprocess_stringlist process_list ValueErrorformat)selfrrQ/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/jiwer/transforms.py__call__Gs    zAbstractTransform.__call__scCs tdSN)NotImplementedErrorrr#rrr r]sz AbstractTransform.process_stringinpcsfdd|DS)Ncsg|]}|qSrr.0r#rrr asz2AbstractTransform.process_list..rrr(rr,r r`szAbstractTransform.process_listN) __name__ __module__ __qualname____doc__rrrr!rrrrrr rBsc@s*eZdZdZeedddZddZdS)ra Chain multiple transformations back-to-back to create a pipeline combining multiple transformations. Note that each transformation needs to end with either `ReduceToListOfListOfWords` or `ReduceToListOfListOfChars`, depending on whether word error rate, or character error rate is desired. Example: ```python3 import jiwer jiwer.Compose([ jiwer.RemoveMultipleSpaces(), jiwer.ReduceToListOfListOfWords() ]) ```  transformscCs ||_dS)zV Args: transforms: The list of transformations to chain. Nr3)rr4rrr __init__xszCompose.__init__cCs|jD] }||}q|Sr$r3)rtexttrrrr r!s  zCompose.__call__N)r/r0r1r2rrr5r!rrrr rdsc@s@eZdZd eedddZedddZeedd d Zd S) BaseRemoveTransform)tokens_to_removecCs||_||_dSr$)r: replace_token)rr:r;rrr r5szBaseRemoveTransform.__init__r"cCs|jD]}|||j}q|Sr$)r:replacer;)rr#wrrr rs z"BaseRemoveTransform.process_stringr'csfdd|DS)Ncsg|]}|qSrr)r*r,rr r-sz4BaseRemoveTransform.process_list..rr.rr,r rsz BaseRemoveTransform.process_listN)r9)r/r0r1rrr5rrrrrr r8sr8c@s@eZdZdZd edddZedddZeed d d Zd S)r a Transforms a single input sentence, or a list of input sentences, into a list with lists of words, which is the expected format for calculating the edit operations between two input sentences on a word-level. A sentence is assumed to be a string, where words are delimited by a token (such as ` `, space). Each string is expected to contain only a single sentence. Empty strings (no output) are removed for the list. Example: ```python import jiwer sentences = ["hi", "this is an example"] print(jiwer.ReduceToListOfListOfWords()(sentences)) # prints: [['hi'], ['this', 'is', 'an, 'example']] ```  word_delimitercCs ||_dS)zo Args: word_delimiter: the character which delimits words. Default is ` ` (space). Nr?rr@rrr r5sz"ReduceToListOfListOfWords.__init__r"cCsdd||jDgS)NcSsg|]}t|dkr|qSlenr+r=rrr r-s z.)splitr@r&rrr rsz(ReduceToListOfListOfWords.process_stringr'cCs<g}|D]}||d}||qt|dkr8ggS|SNrrappendrErr(Zsentence_collectionZsentenceZ list_of_wordsrrr rs  z&ReduceToListOfListOfWords.process_listN)r> r/r0r1r2rr5rrrrrrr r sc@s0eZdZdZedddZeedddZdS) r a Transforms a single input sentence, or a list of input sentences, into a list with lists of characters, which is the expected format for calculating the edit operations between two input sentences on a character-level. A sentence is assumed to be a string. Each string is expected to contain only a single sentence. Example: ```python import jiwer sentences = ["hi", "this is an example"] print(jiwer.ReduceToListOfListOfChars()(sentences)) # prints: [['h', 'i'], ['t', 'h', 'i', 's', ' ', 'i', 's', ' ', 'a', 'n', ' ', 'e', 'x', 'a', 'm', 'p', 'l', 'e']] ``` r"cCsdd|DgS)NcSsg|]}|qSrrrFrrr r-sz.rr&rrr rsz(ReduceToListOfListOfChars.process_stringr'cCs<g}|D]}||d}||qt|dkr8ggS|SrHrIrKrrr rs  z&ReduceToListOfListOfChars.process_listNr/r0r1r2rrrrrrrr r sc@s@eZdZdZd edddZedddZeed d d Zd S)r a& Transforms multiple sentences into a single sentence. This operation can be useful when the number of reference and hypothesis sentences differ, and you want to do a minimal alignment over these lists. Note that this creates an invariance: `wer([a, b], [a, b])` might not be equal to `wer([b, a], [b, a])`. Example: ```python3 import jiwer sentences = ["hi", "this is an example"] print(jiwer.ReduceToSingleSentence()(sentences)) # prints: ['hi this is an example'] ``` r>r?cCs ||_dS)zd :param word_delimiter: the character which delimits words. Default is ` ` (space). Nr?rArrr r5szReduceToSingleSentence.__init__r"cCs|Sr$rr&rrr rsz%ReduceToSingleSentence.process_stringr'cCs6dd|D}t|dkrgSd|j|gSdS)NcSsg|]}t|dkr|qSrBrDr+irrr r-s z7ReduceToSingleSentence.process_list..rz{})rErr@join)rr(Z filtered_inprrr rs z#ReduceToSingleSentence.process_listN)r>rLrrrr r sc@s4eZdZdZeeefdddZedddZdS) rag Transform strings by substituting substrings matching regex expressions into another substring. Example: ```python import jiwer sentences = ["is the world doomed or loved?", "edibles are allegedly cultivated"] # note: the regex string "(\w+)ed", matches every word ending in 'ed', # and "" stands for the first group ('\w+). It therefore removes 'ed' in every match. print(jiwer.SubstituteRegexes({r"doom": r"sacr", r"(\w+)ed": r""})(sentences)) # prints: ["is the world sacr or lov?", "edibles are allegedly cultivat"] ```  substitutionscCs ||_dS)zj Args: substitutions: a mapping of regex expressions to replacement strings. NrQrrRrrr r5 szSubstituteRegexes.__init__r"cCs&|jD]\}}t|||}q |Sr$)rRitemsresubrr#keyvaluerrr r(sz SubstituteRegexes.process_stringNr/r0r1r2rrr5rrrrr r sc@s4eZdZdZeeefdddZedddZdS) ra| This transform can be used to replace a word into another word. Note that the whole word is matched. If the word you're attempting to substitute is a substring of another word it will not be affected. For example, if you're substituting `foo` into `bar`, the word `foobar` will NOT be substituted into `barbar`. Example: ```python import jiwer sentences = ["you're pretty", "your book", "foobar"] print(jiwer.SubstituteWords({"pretty": "awesome", "you": "i", "'re": " am", 'foo': 'bar'})(sentences)) # prints: ["i am awesome", "your book", "foobar"] ``` rQcCs ||_dS)z[ Args: substitutions: A mapping of words to replacement words. NrQrSrrr r5DszSubstituteWords.__init__r"cCs2|jD]"\}}tdt|||}q |S)Nz\b{}\b)rRrTrUrVrescaperWrrr rKszSubstituteWords.process_stringNrZrrrr r/scs*eZdZdZeedfdd ZZS)ra% Can be used to filter out certain words. As words are replaced with a ` ` character, make sure to that `RemoveMultipleSpaces`, `Strip()` and `RemoveEmptyStrings` are present in the composition _after_ `RemoveSpecificWords`. Example: ```python import jiwer sentences = ["yhe awesome", "the apple is not a pear", "yhe"] print(jiwer.RemoveSpecificWords(["yhe", "the", "a"])(sentences)) # prints: [' awesome', ' apple is not pear', ' '] # note the extra spaces ``` )words_to_removecsdd|D}t|dS)zM Args: words_to_remove: List of words to remove. cSsi|] }|dqS)r>r)r+wordrrr jsz0RemoveSpecificWords.__init__..N)superr5)rr\mapping __class__rr r5eszRemoveSpecificWords.__init__)r/r0r1r2rrr5 __classcell__rrrar rRscs(eZdZdZdedfdd ZZS)ra| This transform filters out white space characters. Note that by default space (` `) is also removed, which will make it impossible to split a sentence into a list of words by using `ReduceToListOfListOfWords` or `ReduceToSingleSentence`. This can be prevented by replacing all whitespace with the space character. If so, make sure that `jiwer.RemoveMultipleSpaces`, `Strip()` and `RemoveEmptyStrings` are present in the composition _after_ `RemoveWhiteSpace`. Example: ```python import jiwer sentences = ["this is an example", "hello world "] print(jiwer.RemoveWhiteSpace()(sentences)) # prints: ["thisisanexample", "helloworld"] print(jiwer.RemoveWhiteSpace(replace_by_space=True)(sentences)) # prints: ["this is an example", "hello world "] # note the trailing spaces ``` F)replace_by_spacecs2ddtjD}|rd}nd}tj||ddS)zq Args: replace_by_space: every white space character is replaced with a space (` `) cSsg|]}|qSrr)r+crrr r-sz-RemoveWhiteSpace.__init__..r>r9)r;N)string whitespacer_r5)rrd charactersr;rarr r5s zRemoveWhiteSpace.__init__)F)r/r0r1r2boolr5rcrrrar rosrCcCs$ttjd}tdd|D}|S)z9Compute the punctuation characters only once and memoize.rCcss*|]"}tt|drt|VqdS)PN) unicodedatacategorychr startswithrNrrr sz._get_punctuation_characters..)rangesys maxunicodeset)Z codepoints punctuationrrr _get_punctuation_characterss rucs eZdZdZfddZZS)ra This transform filters out punctuation. The punctuation characters are defined as all unicode characters whose category name starts with `P`. See [here](https://www.unicode.org/reports/tr44/#General_Category_Values) for more information. Example: ```python import jiwer sentences = ["this is an example!", "hello. goodbye"] print(jiwer.RemovePunctuation()(sentences)) # prints: ['this is an example', "hello goodbye"] ``` cst}t|dSr$)rur_r5)rZpunctuation_charactersrarr r5szRemovePunctuation.__init__)r/r0r1r2r5rcrrrar rsc@s0eZdZdZedddZeedddZdS) r ao Filter out multiple spaces between words. Example: ```python import jiwer sentences = ["this is an example ", " hello goodbye ", " "] print(jiwer.RemoveMultipleSpaces()(sentences)) # prints: ['this is an example ', " hello goodbye ", " "] # note that there are still trailing spaces ``` r"cCstdd|S)Nz\s\s+r>rUrVr&rrr rsz#RemoveMultipleSpaces.process_stringr'csfdd|DS)Ncsg|]}|qSrr)r*r,rr r-sz5RemoveMultipleSpaces.process_list..rr.rr,r rsz!RemoveMultipleSpaces.process_listNrMrrrr r sc@seZdZdZedddZdS)ra~ Removes all leading and trailing spaces. Example: ```python import jiwer sentences = [" this is an example ", " hello goodbye ", " "] print(jiwer.Strip()(sentences)) # prints: ['this is an example', "hello goodbye", ""] # note that there is an empty string left behind which might need to be cleaned up ``` r"cCs|Sr$stripr&rrr rszStrip.process_stringNr/r0r1r2rrrrrr rsc@s0eZdZdZedddZeedddZdS) ra  Remove empty strings from a list of strings. Example: ```python import jiwer sentences = ["", "this is an example", " ", " "] print(jiwer.RemoveEmptyStrings()(sentences)) # prints: ['this is an example'] ``` r"cCs|Sr$rwr&rrr rsz!RemoveEmptyStrings.process_stringr'csfdd|DS)Ncsg|]}|dkr|qS)r9r)r*r,rr r-sz3RemoveEmptyStrings.process_list..rr.rr,r rszRemoveEmptyStrings.process_listNrMrrrr rsc@seZdZdZedddZdS)ru6 Replace common contractions such as `let's` to `let us`. Currently, this method will perform the following replacements. Note that `␣` is used to indicate a space (` `) to get around markdown rendering constrains. | Contraction | transformed into | | ------------- |:----------------:| | `won't` | `␣will not` | | `can't` | `␣can not` | | `let's` | `␣let us` | | `n't` | `␣not` | | `'re` | `␣are` | | `'s` | `␣is` | | `'d` | `␣would` | | `'ll` | `␣will` | | `'t` | `␣not` | | `'ve` | `␣have` | | `'m` | `␣am` | Example: ```python import jiwer sentences = ["she'll make sure you can't make it", "let's party!"] print(jiwer.ExpandCommonEnglishContractions()(sentences)) # prints: ["she will make sure you can not make it", "let us party!"] ``` r"cCstdd|}tdd|}tdd|}tdd|}td d |}td d |}td d|}tdd|}tdd|}tdd|}tdd|}|S)Nzwon'tzwill notzcan\'tzcan notzlet\'szlet uszn\'tz notz\'rez arez\'sz isz\'dz wouldz\'llz willz\'tz\'vez havez\'mz amrvr&rrr rsz.ExpandCommonEnglishContractions.process_stringNryrrrr rs c@seZdZdZedddZdS)rz Convert every character into lowercase. Example: ```python import jiwer sentences = ["You're PRETTY"] print(jiwer.ToLowerCase()(sentences)) # prints: ["you're pretty"] ``` r"cCs|Sr$)lowerr&rrr r@szToLowerCase.process_stringNryrrrr r1sc@seZdZdZedddZdS)rz Convert every character to uppercase. Example: ```python import jiwer sentences = ["You're amazing"] print(jiwer.ToUpperCase()(sentences)) # prints: ["YOU'RE AMAZING"] ``` r"cCs|Sr$)upperr&rrr rTszToUpperCase.process_stringNryrrrr rDsc@seZdZdZedddZdS)r a Remove any word between `[]` and `<>`. This can be useful when working with hypotheses from the Kaldi project, which can output non-words such as `[laugh]` and ``. Example: ```python import jiwer sentences = ["you like [laugh]"] print(jiwer.RemoveKaldiNonWords()(sentences)) # prints: ["you like "] # note the extra spaces ``` r"cCstdd|S)Nz[<\[][^>\]]*[>\]]r9rvr&rrr rksz"RemoveKaldiNonWords.process_stringNryrrrr r Xs) r2rq functoolsrUrfrktypingrrr__all__objectrrr8r r r rrrr lru_cacherurr rrrrrr rrrr sX"#-%%"#* 6