U -e$j @sVddlZddlZddlmZddlmZddlmZmZddl m Z m Z m Z m Z mZmZmZddlZddlmZddlmZddlmZdd lmZd d lmZd d lmZd d lmZmZm Z d dl!m"Z"d dl#m$Z$d dl%m&Z&m'Z'd dl(m)Z)m*Z*e+e"j,Z-e$.e/Z0Gddde+Z1Gddde2Z3dZ4e"j,ddge"j5ddddge"j6dddd giZ7d!Z8d"d#gZ9d$d%gZ:e"j,e"j5e"j6gZ;d&d'e;DZe4gZ?e=egZ@d*d+d,d-gZAd.ZBd/d0d1d2d3gZCe+eDd4d5d6ZEee e e+fe e+ee e+d7ffd8d9d:ZFe+e+eDd;dd?ZHe e+ge e+fe e+e e+fd@dAdBZIe e+ge e+fe e+d@dCdDZJdWe+e+e e e+e ee e+dEdFdGZKdXe+e ee e+e e+fdHdIdJZLdYe+e ee e+dHdKdLZMdZe+e eee+dMdNdOZNd[e e+e eee+dQdRdSZOGdTd7d7e e+ZPGdUdVdVe e+ePfZQdS)\N)partial) has_magic)PathPurePath)CallableDictListOptionalSetTupleUnion)get_fs_token_paths)HTTPFileSystem) HfFileSystem) thread_map)config)DownloadConfig)!_prepare_path_and_storage_options xbasenamexjoin)Split)logging) is_local_pathis_relative_path)glob_pattern_to_regexstring_to_dictc@s eZdZdS)UrlN__name__ __module__ __qualname__r"r"T/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/data_files.pyrsrc@s eZdZdS)EmptyDatasetErrorNrr"r"r"r#r$!sr$zFdata/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*trainZtrainingZ validationZvaliddevvaltesttestingevalZ evaluationz-._ 0-9z**[{sep}/]{keyword}[{sep}]*z{keyword}[{sep}]*z{keyword}[{sep}/]**z**[{sep}/]{keyword}[{sep}/]**cCs i|]}|ddt|DqS)cSs$g|]}tD]}|j|tdq qS)keywordsep)"KEYWORDS_IN_FILENAME_BASE_PATTERNSformatNON_WORDS_CHARS.0r,patternr"r"r# 2s.SPLIT_KEYWORDSr2splitr"r"r# 1s r:cCs i|]}|ddt|DqS)cSs$g|]}tD]}|j|tdq qSr+)"KEYWORDS_IN_DIR_NAME_BASE_PATTERNSr/r0r1r"r"r#r4:sr5r6r8r"r"r#r:9s z**z metadata.csvz**/metadata.csvzmetadata.jsonlz**/metadata.jsonlz*[]z README.mdz config.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.json)r3returncstfddtDS)Nc3s|]}|kVqdSNr")r2Zwilcard_characterr3r"r# Wsz%contains_wildcards..)anyWILDCARD_CHARACTERSr>r"r>r#contains_wildcardsVsrB DataFilesList)patternsr<cCst|trdd|DSt|tr0t|giSt|trtdd|Dr|D]D}t|trt|dkrd|krt|dttfsPt d|qPd d |D}tt |t|krt d |d d|DSt|iSn t t|Sd S)a/ Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list of patterns cSs*i|]"\}}t|t|tr |n|gqSr"str isinstancelist)r2keyvaluer"r"r#r:dsz%sanitize_patterns..css|]}t|tVqdSr=)rGdictr2r3r"r"r#r?hsz$sanitize_patterns..r9pathz]Expected each split to have a 'path' key which can be a string or a list of strings, but got cSsg|] }|dqSr9r"rLr"r"r#r4ssz%sanitize_patterns..z*Some splits are duplicated in data_files: cSs6i|].}t|dt|dtr(|dn|dgqS)r9rNrErLr"r"r#r:vs N) rGrKitemsrFSANITIZED_DEFAULT_SPLITrHr@lenget ValueErrorsetsanitize_patterns)rDr3splitsr"r"r#rVZs4      rV)matched_rel_pathr3r<cCs<ddt|jjD}ddt|jjD}t|t|kS)u When a path matches a pattern, we additionnally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False cSsg|]}|dr|qS__ startswithr2partr"r"r#r4s z6_is_inside_unrequested_special_dir..cSsg|]}|dr|qSrYr[r]r"r"r#r4s )rparentpartsrR)rXr3Zdata_dirs_to_ignore_in_pathZdata_dirs_to_ignore_in_patternr"r"r#"_is_inside_unrequested_special_dirsracCs8ddt|jD}ddt|jD}t|t|kS)u: When a path matches a pattern, we additionnally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False cSs(g|] }|drt|dhks|qS.r\rUr]r"r"r#r4s zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..cSs(g|] }|drt|dhks|qSrbrdr]r"r"r#r4s )rr`rR)rXr3Zhidden_directories_in_pathZhidden_directories_in_patternr"r"r#?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirs5re)pattern_resolverr<c s.tD]dd}z ||}Wntk r8YqYnXt|dkrfdd|DfddtDttt}fdd |DSqtD]~g}D]R\}}|D]D}z ||}Wntk rYqYnXt|dkr| |qqq|rfd d |DSqtd |d |d S)a+ Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rcsh|]}t|tdqSrO)rr)r2p split_patternr"r# sz+_get_data_files_patterns..csg|]}|krt|qSr")rFr8)rWr"r#r4sz,_get_data_files_patterns..csi|]}|j|dgqS)rO)r/r8rir"r#r:sz,_get_data_files_patterns..csi|]}||qSr"r"r8) patterns_dictr"r#r:sCouldn't resolve pattern  with resolver N) ALL_SPLIT_PATTERNSreplaceFileNotFoundErrorrRDEFAULT_SPLITSsortedrUALL_DEFAULT_PATTERNSrPappend)rfr3 data_filesZ sorted_splitsZnon_empty_splitsr9rDr")rlrjrWr#_get_data_files_patternss4         rwc Csfg}tD]<}z"||}t|dkr,||Wqtk rBYqXq|rN|Std|d|dS)zM Get the supported metadata patterns from a directory or repository. rrmrnN)METADATA_PATTERNSrRrurq)rfZnon_empty_patternsr3Zmetadata_filesr"r"r#_get_metadata_files_patternss ry)r3 base_pathallowed_extensionsdownload_configr<c st|rt||}n$t|r4tj|dtj}nd}t||d\}}t||d\}}}| dd ddpv|j | dd ddt t t |ht|jtr|jn|jd}|dkr|dndfd d |j|d d D}d k rRfdd |D} t| t|krVtt |t | } td|d| n|} | sd|d} d k r| dt7} t| | S)a Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. Here are some behaviors specific to fsspec glob that are different from glob.glob, Path.glob, Path.match or fnmatch: - '*' matches only first level items - '**' matches all items - '**/*' matches all at least second level items More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicilty mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rr|storage_optionsz::z://filecszg|]r\}}|ddkrt|krttj|tjsttj|tjs|rn|n|qS)typer)rraosrNrelpathrer\)r2filepathinfo)files_to_ignore fs_base_path fs_patternprotocol_prefixr"r#r4Ss    z#resolve_pattern..T)detailNcs8g|]0}tfddt|dddDr|qS)c3s|]}d|kVqdS)rcNr")r2suffixr{r"r#r?csz-resolve_pattern...rcrN)r@rr9)r2rrr"r#r4`s(z Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rrrrrN splitdriver-rr r9Z root_markerrUFILES_TO_IGNORErrGprotocolrFglobrPrRrHloggerrrq) r3rzr{r|rfs_rZ matched_pathsoutZinvalid_matched_files error_msgr")r{rrrrr#resolve_patterns>1     r)rzr|r<cCsDtt||d}z t|WStk r>td|ddYnXdS)u Get the default pattern from a directory testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. Some examples of supported patterns: Input: my_dataset_repository/ ├── README.md └── dataset.csv Output: {"train": ["**"]} Input: my_dataset_repository/ ├── README.md ├── train.csv └── test.csv my_dataset_repository/ ├── README.md └── data/ ├── train.csv └── test.csv my_dataset_repository/ ├── README.md ├── train_0.csv ├── train_1.csv ├── train_2.csv ├── train_3.csv ├── test_0.csv └── test_1.csv Output: {"train": ["**train*"], "test": ["**test*"]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train/ │ ├── shard_0.csv │ ├── shard_1.csv │ ├── shard_2.csv │ └── shard_3.csv └── test/ ├── shard_0.csv └── shard_1.csv Output: {"train": ["**train*/**"], "test": ["**test*/**"]} Input: my_dataset_repository/ ├── README.md └── data/ ├── train-00000-of-00003.csv ├── train-00001-of-00003.csv ├── train-00002-of-00003.csv ├── test-00000-of-00001.csv ├── random-00000-of-00003.csv ├── random-00001-of-00003.csv └── random-00002-of-00003.csv Output: { "train": ["data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"], "test": ["data/test-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"], "random": ["data/random-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*"], } In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. rzr|The directory at z doesn't contain any data filesN)rrrwrqr$rzr|resolverr"r"r#get_data_patternsts T rcCsDtt||d}z t|WStk r>td|ddYnXdS)zE Get the supported metadata patterns from a local directory. rrz" doesn't contain any metadata fileN)rrryrqrr"r"r#get_metadata_patternss  r) data_filer|r<c Cst||d\}}t||d\}}}t|trB||}|j|jfSt|tr|t j rtt j |j d}d|t t j dd ddd}||}|j|jfS||}dD]}||krt||fSqd S) Nr~r)Zendpointtokenzhf://rz /resolve/@)ETagetagmtimer")rr rGrZ resolve_pathZrepo_idrevisionrr\rZ HF_ENDPOINTrrRrprrF) rr|rrr resolved_pathZhffsrrIr"r"r#_get_single_origin_metadatas   $   r@)rvr|r<cCs0ttt|d||tjdt|dkp*t dS)Nr~zResolving data files) max_workersZ tqdm_classdescdisable)rrrrZtqdmrRZis_progress_bar_enabled)rvrr|r"r"r#_get_origin_metadatas rc seZdZdZeeeeedfdd ZddZe deee j j e ee eee eddd d Ze deee ee eee edd d d Ze deee ee eee edd ddZeeddddZZS)rCa List of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover DataFilesList has an additional attribute ``origin_metadata``. It can store: - the last modified time of local files - ETag of remote files - commit sha of a dataset repository Thanks to this additional attribute, it is possible to hash the list and get a different hash if and only if at least one file changed. This is useful for caching Dataset objects that are obtained from a list of data files. )rvorigin_metadatacst|||_dSr=)super__init__r)selfrvr __class__r"r#rs zDataFilesList.__init__cCst|||j|jSr=)rCr)rotherr"r"r#__add__szDataFilesList.__add__NrD dataset_inforzr{r|r<cCs6d|jd|jd|pdd}|j||||dS)Nzhf://datasets/r/r}rzr{r|)idsharstrip from_patterns)clsrDrrzr{r|r"r"r# from_hf_repos $zDataFilesList.from_hf_reporDrzr{r|r<cCs,|dk r |n t}|j||||dSNr)rresolveas_posixr)rrDrzr{r|r"r"r#from_local_or_remote)sz"DataFilesList.from_local_or_remotec Csx|dk r |n t}g}|D]>}z|t||||dWq"tk r^t|sZYq"Xq"t||d}|||S)Nrr~)rrrextendrrqrr)rrDrzr{r|rvr3rr"r"r#r6s"  zDataFilesList.from_patterns extensionsr<csBddd|Dtddtfdd|D|jdS) N|css|]}d|VqdS)\Nr")r2extr"r"r#r?Qsz2DataFilesList.filter_extensions..z.*(z )(\..+)?$csg|]}|r|qSr")match)r2rr>r"r#r4Ts z3DataFilesList.filter_extensions..)r)joinrecompilerCr)rrr"r>r#filter_extensionsPs zDataFilesList.filter_extensions)NNN)NNN)NNN)rr r!__doc__rrFr rr classmethodhuggingface_hubhf_api DatasetInfor rrrrr __classcell__r"r"rr#rCsL      c @seZdZdZedeeeeee ffe ee eee e ddddZ edeeeeee ffe jje ee eee e ddddZedeeeeee ffe ee eee e ddd d Zeedd d d ZdS) DataFilesDicta Dict of split_name -> list of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover each list is a DataFilesList. It is possible to hash the dictionary and get a different hash if and only if at least one file changed. For more info, see ``DataFilesList``. This is useful for caching Dataset objects that are obtained from a list of data files. Changing the order of the keys of this dictionary also doesn't change its hash. NrcCs@|}|D],\}}t|ts2tj||||dn|||<q|Sr)rPrGrCrrrDrzr{r|rrIpatterns_for_keyr"r"r#ris z"DataFilesDict.from_local_or_remoterc CsB|}|D].\}}t|ts4tj|||||dn|||<q|S)N)rrzr{r|)rPrGrCr) rrDrrzr{r|rrIrr"r"r#rs   zDataFilesDict.from_hf_repocCs@|}|D],\}}t|ts2tj||||dn|||<q|Sr)rPrGrCrrr"r"r#rs zDataFilesDict.from_patternsrcCs.t|}|D]\}}||||<q|Sr=)rrPr)rrrrIZdata_files_listr"r"r#rs zDataFilesDict.filter_extensions)NNN)NNN)NNN)rr r!rrrrFr rrCr rrrrrrrrr"r"r"r#rYsH   r)NN)N)N)N)rN)Rrr functoolsrrrpathlibrrtypingrrrr r r r rZfsspecr Zfsspec.implementations.httprrZtqdm.contrib.concurrentrr}rdownloadrZ#download.streaming_download_managerrrrrWrutilsrZutils.file_utilsrrZutils.py_utilsrrrFZTRAINrQZ get_loggerrrrrqr$ZSPLIT_PATTERN_SHARDEDZ VALIDATIONZTESTr7r0r.r;rrZ"DEFAULT_PATTERNS_SPLIT_IN_FILENAMEZ"DEFAULT_PATTERNS_SPLIT_IN_DIR_NAMEZDEFAULT_PATTERNS_ALLrortrxrArboolrBrVrarerwryrrrrrrCrr"r"r"r#s  $             .&!>*%"  _$]   W