U -e@stddlmZmZddlmZmZmZmZmZm Z m Z ddl Z erLddl mZeGdddZeGdd d ZdS) ) dataclassfield) TYPE_CHECKINGAnyClassVarDictListOptionalUnionN) FeatureTypec@seZdZUdZeeed<dZeeed<dZ e eed<dZ e e ed<e dddd Zeed <d d Zed eed ffdddZdS) Translationa`FeatureConnector` for translations with fixed languages per example. Here for compatiblity with tfds. Args: languages (`dict`): A dictionary for each example mapping string language codes to string translations. Example: ```python >>> # At construction time: >>> datasets.features.Translation(languages=['en', 'fr', 'de']) >>> # During data generation: >>> yield { ... 'en': 'the cat', ... 'fr': 'le chat', ... 'de': 'die katze' ... } ``` languagesNiddictdtypepa_typeFdefaultinitrepr_typecCstddt|jDS)NcSsi|]}|tqS)pastring).0langrr^/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/features/translation.py *sz(Translation.__call__..)rstructsortedrselfrrr__call__)szTranslation.__call__r returncs$ddlmfddt|jDS)z2Flatten the Translation feature into a dictionary.r Valuecsi|]}|dqS)rr)rkr&rrr0sz'Translation.flatten..)featuresr'r rr!rr&rflatten,s zTranslation.flatten)__name__ __module__ __qualname____doc__rstr__annotations__rr rrrrrrr#r rr*rrrrr s  r c@seZdZUdZdZeeed<dZee ed<dZ ee ed<dZ e e ed<dZe eed<edd d d Ze ed <d d ZddZddZedee dffdddZdS)TranslationVariableLanguagesa$`FeatureConnector` for translations with variable languages per example. Here for compatiblity with tfds. Args: languages (`dict`): A dictionary for each example mapping string language codes to one or more string translations. The languages present may vary from example to example. Returns: - `language` or `translation` (variable-length 1D `tf.Tensor` of `tf.string`): Language codes sorted in ascending order or plain text translations, sorted to align with language codes. Example: ```python >>> # At construction time: >>> datasets.features.TranslationVariableLanguages(languages=['en', 'fr', 'de']) >>> # During data generation: >>> yield { ... 'en': 'the cat', ... 'fr': ['le chat', 'la chatte,'] ... 'de': 'die katze' ... } >>> # Tensor returned : >>> { ... 'language': ['en', 'de', 'fr', 'fr'], ... 'translation': ['the cat', 'die katze', 'la chatte', 'le chat'], ... } ``` Nr num_languagesrrrrFrrcCs4|jrtt|jnd|_|jr*t|jnd|_dS)N)rr setlenr2r!rrr __post_init__\sz*TranslationVariableLanguages.__post_init__cCs$tttttdS)Nlanguage translation)rrlist_rr!rrrr#`sz%TranslationVariableLanguages.__call__cst|j}|jrJt||rJtddtt||dd|dg}|D]:\}t|trx||fqV| fdd|DqVt t|\}}||dS)NzSome languages in example (z, z) are not in valid set (z).csg|] }|fqSrr)relrrr qsz?TranslationVariableLanguages.encode_example..r6) r3r ValueErrorjoinr items isinstancer/appendextendzip)r"Ztranslation_dictZlang_setZtranslation_tuplestextrZ translationsrr;rencode_examplecs ( z+TranslationVariableLanguages.encode_exampler r$cCs*ddlm}m}||d||ddS)zCFlatten the TranslationVariableLanguages feature into a dictionary.r )Sequencer'rr6)r)rFr')r"rFr'rrrr*xs  z$TranslationVariableLanguages.flatten)r+r,r-r.rr rr0r2intrr/rrrrrrr5r#rEr rr*rrrrr13s r1) dataclassesrrtypingrrrrrr r Zpyarrowrr)r r r1rrrrs$ '