U -eh@sdZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZmZddlZddlmZmZddlmZdd lmZmZdd lmZdd lmZdd lmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&e#e'Z(eGdddZ)eGdddZ*Gddde+Z,Gddde+Z-eGdddZ.eGdddZ/Gddde e0e/fZ1eGdddZ2dS) a DatasetInfo and MetricInfo record information we know about a dataset and a metric. This includes things that we know about the dataset statically, i.e.: - description - canonical location - does it have validation and tests splits - size - etc. This also includes the things that can and should be computed once we've processed the dataset as well: - number of examples (in each split) - etc. N) dataclass)Path)ClassVarDictListOptionalUnion) DatasetCardDatasetCardData)config)FeaturesValue)is_remote_filesystem) SplitDict) TaskTemplatetask_template_from_dict)Version) get_logger)asdict unique_valuesc@s&eZdZUdZeed<dZeed<dS)SupervisedKeysDatainputoutputN)__name__ __module__ __qualname__rstr__annotations__rr r N/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/info.pyr9s  rc@s&eZdZUdZeed<dZeed<dS)DownloadChecksumsEntryDatarkeyvalueN)rrrr#rrr$r r r r!r"?s  r"c@seZdZdZdS)MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nrrr__doc__r r r r!r%Esr%c@seZdZdZdS)NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr&r r r r!r(Isr(c@sJeZdZUdZeeed<dZeeed<ddZ e eddddZ dS) PostProcessedInfoNfeaturesresources_checksumscCs(|jdk r$t|jts$t|j|_dSN)r* isinstancer from_dictselfr r r! __post_init__RszPostProcessedInfo.__post_init__)post_processed_info_dictreturncs0ddt|D|ffdd|DS)NcSsh|] }|jqSr name.0fr r r! Ysz.PostProcessedInfo.from_dict..csi|]\}}|kr||qSr r r7kv field_namesr r! Zsz/PostProcessedInfo.from_dict.. dataclassesfieldsitems)clsr2r r=r!r.WszPostProcessedInfo.from_dict) rrrr*rr rr+dictr1 classmethodr.r r r r!r)Ms r)c@s eZdZUdZejedZeed<ejedZ eed<ejedZ eed<ejedZ eed<dZ e eed<dZe eed <dZe eed <dZe eeed <dZe eed <dZe eed <dZe eed<dZe eeefed<dZe eed<dZe eed<dZe e ed<dZ!e e ed<dZ"e e ed<dZ#e e ed<dddddgZ$e%eeed<ddZ&d7e edddZ'd8ddZ(d d!Z)e*edd"d#d$Z+e*d9ee edd%d&d'Z,e*edd(d)d*Z-d:dd,d-d.Z.dd/d0d1Z/ed/d2d3Z0e*edd4d5d6Z1dS); DatasetInfoa Information about a dataset. `DatasetInfo` documents datasets, including its name, version, and features. See the constructor arguments and properties for a full list. Not all fields are known on construction and may be updated later. Attributes: description (`str`): A description of the dataset. citation (`str`): A BibTeX citation of the dataset. homepage (`str`): A URL to the official homepage for the dataset. license (`str`): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. features ([`Features`], *optional*): The features used to specify the dataset's column types. post_processed (`PostProcessedInfo`, *optional*): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index. supervised_keys (`SupervisedKeysData`, *optional*): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). builder_name (`str`, *optional*): The name of the `GeneratorBasedBuilder` subclass used to create the dataset. Usually matched to the corresponding script name. It is also the snake_case version of the dataset builder class name. config_name (`str`, *optional*): The name of the configuration derived from [`BuilderConfig`]. version (`str` or [`Version`], *optional*): The version of the dataset. splits (`dict`, *optional*): The mapping between split name and metadata. download_checksums (`dict`, *optional*): The mapping between the URL to download the dataset's checksums and corresponding metadata. download_size (`int`, *optional*): The size of the files to download to generate the dataset, in bytes. post_processing_size (`int`, *optional*): Size of the dataset in bytes after post-processing, if any. dataset_size (`int`, *optional*): The combined size in bytes of the Arrow tables for all splits. size_in_bytes (`int`, *optional*): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files). task_templates (`List[TaskTemplate]`, *optional*): The task templates to prepare the dataset for during training and evaluation. Each template casts the dataset's [`Features`] to standardized column names and types as detailed in `datasets.tasks`. **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`]. default_factory descriptioncitationhomepagelicenseNr*post_processedsupervised_keystask_templates builder_name dataset_name config_nameversionsplitsdownload_checksums download_sizepost_processing_size dataset_size size_in_bytes_INCLUDED_INFO_IN_YAMLcsjdk r$tjts$tj_jdk rHtjtsHtj_jdk rtjtstjtrxtj_ntj_j dk rtj t st j _ j dk rtj t stj ttfrt j _ nt fj _ jdk rftjttfr,ddjD}dd|D_n:tjtrFjg_n tj}|dk r`|gng_jdk rtj_jdk rfddjD_dS)NcSs"g|]}t|tr|nt|qSr )r-rrr7templater r r! sz-DatasetInfo.__post_init__..cSsg|]}|dk r|qSr,r r\r r r!r^scsg|]}|jqSr )Zalign_with_featuresr*r\r/r r!r^s)r*r-r r.rNr)rTrrrUrZfrom_split_dictrOrtuplelistrPrr)r0Z templatesr]r r/r!r1s<        zDatasetInfo.__post_init__F deprecatedstorage_optionsc Cs|dkrtdt|j}tj||d}|d}t| }|rFtjj nt j }| ||t j d}|j||dW5QRX|jr| ||t jd}||W5QRXdS)ahWrite `DatasetInfo` and license (if present) as JSON files to `dataset_info_dir`. Args: dataset_info_dir (`str`): Destination directory. pretty_print (`bool`, defaults to `False`): If `True`, the JSON will be pretty-printed with the indent level of 4. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("rotten_tomatoes", split="validation") >>> ds.info.write_to_directory("/path/to/directory/") ``` ra'fs' was deprecated in favor of 'storage_options' in version 2.9.0 and will be removed in 3.0.0. You can remove this warning by passing 'storage_options=fs.storage_options' instead.rbrwb) pretty_printN)warningswarn FutureWarningrcfsspecget_fs_token_pathsrospathjoin posixpathopenr DATASET_INFO_FILENAME _dump_inforMLICENSE_FILENAME _dump_license) r0dataset_info_dirrffsrcfs_token_pathsis_local path_joinr8r r r!write_to_directorys! zDatasetInfo.write_to_directorycCs*|tjt||rdnddddS)zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsrencode)r0filerfr r r!rr szDatasetInfo._dump_infocCs||jddS)zTDump license in `file` file-like object open in bytes mode (to support remote files)r~N)rrMr)r0rr r r!rtszDatasetInfo._dump_license) dataset_infosc Csdd|D}dtdd|D}dtdd|D}dtdd|D}dtdd|D}d}d}d}d d|D} t| d krtt| d j| d d}nt| rtt| d }|r|nd}||||||||d S) NcSsg|]}|dk r|qSr,)copy)r7 dset_infor r r!r^sz*DatasetInfo.from_merge..z css|] }|jVqdSr,)rJr7infor r r! sz)DatasetInfo.from_merge..css|] }|jVqdSr,)rKrr r r!rscss|] }|jVqdSr,)rLrr r r!rscss|] }|jVqdSr,)rMrr r r!rscSsg|]}|jdk r|jqSr,)rPrr r r!r^s r r)rJrKrLrMr*rOrP)rnrstriplenr`set intersection) rDrrJrKrLrMr*rOrPZall_task_templatesr r r! from_merges.   zDatasetInfo.from_merge)rurcr3c Cs|dkrtdt|j}tj||d}|d}td||sLtdt | }|rbt j j nt j }|j||tjddd }t|}W5QRX||S) aCreate [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_dir (`str`): The directory containing the metadata file. This should be the root directory of a specific dataset version. fs (`fsspec.spec.AbstractFileSystem`, *optional*): Instance of the remote filesystem used to download the files from. `fs` was deprecated in version 2.9.0 and will be removed in 3.0.0. Please use `storage_options` instead, e.g. `storage_options=fs.storage_options`. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import DatasetInfo >>> ds_info = DatasetInfo.from_directory("/path/to/directory/") ``` rardrbrzLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rr~encoding)rgrhrircrjrkloggerr ValueErrorrrlrmrnrorpr rqrloadr.) rDrurvrcrwrxryr8dataset_info_dictr r r!from_directory0s % zDatasetInfo.from_directory)rr3cs0ddt|D|ffdd|DS)NcSsh|] }|jqSr r4r6r r r!r9msz(DatasetInfo.from_dict..csi|]\}}|kr||qSr r r:r=r r!r?nsz)DatasetInfo.from_dict..r@)rDrr r=r!r.kszDatasetInfo.from_dictT)other_dataset_infocs*|j}|jffdd|jDdS)Ncs(i|] \}}|dk ss|t|qSr,rdeepcopyr: ignore_noner r!r?ss z&DatasetInfo.update..)__dict__updaterC)r0rrZ self_dictr rr!rps  zDatasetInfo.updater3cCs|jfdd|jDS)NcSsi|]\}}|t|qSr rr:r r r!r?{sz$DatasetInfo.copy..) __class__rrCr/r r r!rzszDatasetInfo.copycCsfi}t|}|D]P}||jkrt||}t|dr@|||<qt|drX|||<q|||<q|S)N _to_yaml_list_to_yaml_string)rr[getattrhasattrrr)r0Z yaml_dictrr#r$r r r! _to_yaml_dict}s     zDatasetInfo._to_yaml_dict) yaml_datar3cszt|}|ddk r*t|d|d<|ddk rJt|d|d<ddt|D|ffdd|DS)Nr*rUcSsh|] }|jqSr r4r6r r r!r9sz.DatasetInfo._from_yaml_dict..csi|]\}}|kr||qSr r r:r=r r!r?sz/DatasetInfo._from_yaml_dict..) rrgetr Z_from_yaml_listrrArBrC)rDrr r=r!_from_yaml_dicts zDatasetInfo._from_yaml_dict)FraN)F)raN)T)2rrrr'rAfieldrrJrrKrLrMr*rr rNr)rOrrPrrrQrRrSrTrrrUrErVrWintrXrYrZr[rr1rzrrrtrFrrr.rrrrr r r r!rG]sf /* 5 : rGc@sReZdZd ddddZeddddZeeddd d Zeddd d ZdS)DatasetInfosDictFNrc Csi}tj|tj}tj|d}|s0||}||tj|rt|ddd0}dd| D}t j |||rxdnddW5QRXtj|rt |} | j} n d} t} |r|| | dkrt d t| d n| } | t|dS) N README.mdwr~rcSsi|]\}}|t|qSr )rr7rSrr r r!r?sz7DatasetInfosDict.write_to_directory..r{r|z--- z --- )rlrmrnr DATASETDICT_INFOS_FILENAMErrexistsrprCrdumpr rdatar to_dataset_card_datarsaver) r0dataset_infos_dir overwriterftotal_dataset_infosZdataset_infos_pathZdataset_readme_pathr8Zdataset_infos_dictZ dataset_carddataset_card_datar r r!rzs,   "   z#DatasetInfosDict.write_to_directoryc Cstd|tjtj|drLtt|dj }d|krL| |Stjtj|t j rt tj|t j dd*}|ddt|DW5QRSQRXn|SdS)NzLoading Dataset Infos from r dataset_infor~rcSsi|]\}}|t|qSr )rGr.)r7rSrr r r!r?sz3DatasetInfosDict.from_directory..)rrrlrmrrnr rrrfrom_dataset_card_datar rrprrC)rDrrr8r r r!rs  zDatasetInfosDict.from_directory)rr3cCsrt|dttfrht|dtr8|dd|dDSt|d}|ddd|_||j|iSn|SdS)NrcSs i|]}|ddt|qS)rSdefault)rrGr)r7dataset_info_yaml_dictr r r!r?s  z;DatasetInfosDict.from_dataset_card_data..rSr)r-rr`rErGrrS)rDrrr r r!rs z'DatasetInfosDict.from_dataset_card_datacCs*|r&d|kr6t|dtr6|ddd|di}n.d|kr`t|dtr`dd|dD}ni}|dd|D}|D]\}}||d<qt|dkrtt||d<|d dd}|dkrd|i|d|d<nDg|d<t |D].\}}| ddd|i|}|d |qdS)NrrSrcSsi|]}|d|qS)rSr )r7Zconfig_metadatar r r!r?sz9DatasetInfosDict.to_dataset_card_data..cSsi|]\}}||qSr )rrr r r!r?sr ) r-rErr`rCrnextitervaluespopsortedappend)r0rZdataset_metadata_infosrrSZdset_info_yaml_dictrr r r!rs:     z%DatasetInfosDict.to_dataset_card_data)FF) rrrrzrFrr rrr r r r!rs rc@seZdZUdZeed<eed<eed<ejedZ eed<ejedZ eed<ejedZ eed<eje dZ eeed <eje dZeeed <d Zeed <d Zeeed<d Zeeed<d Zeeed<d Zeeed<ddZdddZeddddZeeddddZd S) MetricInfoaInformation about a metric. `MetricInfo` documents a metric, including its name, version, and features. See the constructor arguments and properties for a full list. Note: Not all fields are known on construction and may be updated later. rJrKr*rHinputs_descriptionrLrM codebase_urlsreference_urlsF streamableNformat metric_namerS experiment_idcCsD|jdk r@|jD]*\}}t|tstd|d|jjqdS)NzSWhen using 'numpy' format, all features should be a `datasets.Value` feature. Here z is an instance of )rr*rCr-rrrr)r0r#r$r r r!r1s   zMetricInfo.__post_init__c Cs~ttj|tjddd"}tjt|||r0dnddW5QRX|j rzttj|tj ddd}| |j W5QRXdS)aWrite `MetricInfo` as JSON to `metric_info_dir`. Also save the license separately in LICENCE. If `pretty_print` is True, the JSON will be pretty-printed with the indent level of 4. Example: ```py >>> from datasets import load_metric >>> metric = load_metric("accuracy") >>> metric.info.write_to_directory("/path/to/directory/") ``` rr~rr{Nr|) rprlrmrnr METRIC_INFO_FILENAMErrrrMrsr)r0metric_info_dirrfr8r r r!rz's &zMetricInfo.write_to_directoryrc CsTtd||stdttj|tjdd}t |}W5QRX| |S)aCreate MetricInfo from the JSON file in `metric_info_dir`. Args: metric_info_dir: `str` The directory containing the metadata file. This should be the root directory of a specific dataset version. Example: ```py >>> from datasets import MetricInfo >>> metric_info = MetricInfo.from_directory("/path/to/directory/") ``` zLoading Metric info from zCCalling MetricInfo.from_directory() with undefined metric_info_dir.r~r) rrrrprlrmrnr rrrr.)rDrr8metric_info_dictr r r!r;s zMetricInfo.from_directory)rr3cs0ddt|D|ffdd|DS)NcSsh|] }|jqSr r4r6r r r!r9Tsz'MetricInfo.from_dict..csi|]\}}|kr||qSr r r:r=r r!r?Usz(MetricInfo.from_dict..r@)rDrr r=r!r.RszMetricInfo.from_dict)F)rrrr'rrr rArrrLrMr`rrrrboolrrrrSrr1rzrFrrEr.r r r r!rs(    r)3r'rrArrlrorgrpathlibrtypingrrrrrrjZhuggingface_hubr r rr r*r rZ filesystemsrrUrtasksrrutilsrZ utils.loggingrZutils.py_utilsrrrrrr" Exceptionr%r(r)rGrrrr r r r!sF       :m