U -e(i@sdZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z m Z mZddlZddlmZddlmZddlmZmZdd lmZmZmZmZdd lmZdd l m!Z!e rdd l"m#Z#dd l$m%Z%m&Z&e'e(Z)dZ*e+dedddej,Z-e+dZ.Gddde/Z0Gddde/Z1eddGdddZ2d3e3e dee3dfe e3e e3e2dddZ4Gd d!d!Z5Gd"d#d#e5Z6Gd$d%d%e5Z7eddGd&d'd'Z8eddGd(d)d)Z9d*d+Z:d,d-Z;d.d/ZdS)4z Arrow ArrowReader.N) dataclass)Path) TYPE_CHECKINGListOptionalUnion)DownloadConfig) _split_refilenames_for_dataset_split) InMemoryTableMemoryMappedTableTable concat_tables)logging) cached_path) DatasetInfo)Split SplitInfoz=https://storage.googleapis.com/huggingface-nlp/cache/datasetsz ^ (?Pz) (\[ ((?P-?\d+) (?P%)?)? : ((?P-?\d+) (?P%)?)? \])?(\((?P[^\)]*)\))? $ z\s*\+\s*c@seZdZdZdS)DatasetNotOnHfGcsErrorz?When you can't get the dataset from the Hf google cloud storageN__name__ __module__ __qualname____doc__rrV/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/arrow_reader.pyrAsrc@seZdZdZdS)MissingFilesOnHfGcsErrorz9When some files are missing on the Hf oogle cloud storageNrrrrrrGsrT)frozenc@s&eZdZUdZeed<eeed<dS)FileInstructionsa}The file instructions associated with a split ReadInstruction. Attributes: num_examples: `int`, The total number of examples file_instructions: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.slice(skip, take)` num_examplesfile_instructionsN)rrrrint__annotations__rdictrrrrr Ms  r rReadInstruction)name split_infos instructionfiletype_suffix prefix_pathreturncstts tdtjn s,tddd|D}dd|Dfdd|D}t|tstt|}||}g}d} |D].} || j } || j } | j } | j dkrdn| j }| j dkr| n| j }| dkr| D]&}| ||7} | ||||d qqd}d}t | | D]\}}||7}||kr||kr||krV||nd}||krp|||nd }|dkrq | |||d | |d kr||n|7} ||7}q qt| |d S) aReturns instructions of the split dict. Args: name (`str`): Name of the dataset. split_infos (`list` of `[SplitInfo]`): Dataset splits information. instruction ([`ReadInstruction`] or `str`): Reading instruction for a dataset. filetype_suffix (`str`, *optional*): Suffix of dataset files, e.g. 'arrow' or 'parquet'. prefix_path (`str`, *optional*): Prefix of dataset files, e.g. directory name. Returns: [`FileInstructions`] zExpected str 'name', but got: zExpected non-empty str 'name'cSsi|]}|j|jqSr)r'r!.0inforrr ssz*make_file_instructions..cSsi|]}|j|jqSr)r' shard_lengthsr-rrrr0tsc s*i|]"}|jt|j|jdqS))pathZ dataset_namesplitr*r1)r'r r-r*r'Zname2shard_lengthsr+rrr0usrN)filenameskiptaker)r!r") isinstancestr TypeErrortyper ValueErrorr& from_spec to_absolute splitnamefrom_toappendzipr )r'r(r)r*r+name2lenZname2filenamesZabsolute_instructionsr"r!Z abs_instrZ split_length filenamesr1r@rAr5Z index_startZ index_endZ shard_lengthr6r7rr4rmake_file_instructions\sP           rFc@s~eZdZdZeeddddZdeddd Zdedd d Z d d Z dddZ de e eddddZedddZdS) BaseReaderz@ Build a Dataset object out of Instruction instance(s). rr2r/cCs||_||_d|_dS)zInitializes ArrowReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. N)_path_info_filetype_suffixselfr2r/rrr__init__szBaseReader.__init__Fr,cCstdS)=Returns a Dataset instance from given (filename, skip, take).N)NotImplementedError)rMfilename_skip_take in_memoryrrr_get_table_from_filenamesz#BaseReader._get_table_from_filenamecCst|dkstdd|Ds&tdg}t|}|D]}tj|j|d|d<q8|D]}|j ||d}| |qZdd|D}|s|j d ks|j j d krtd |pt jgt|j j jd g}t|d krt|n|d}|S) aReturns Dataset for given file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contain the absolute path, not relative. skip/take indicates which example read in the file: `ds.slice(skip, take)` in_memory (bool, default False): Whether to copy the data in-memory. rcss|]}t|tVqdSN)r8r%)r.frrr sz)BaseReader._read_files..z&please provide valid file informationsr5rScSsg|]}t|dkr|qS)r)len)r.trrr s z*BaseReader._read_files..NzqTried to read an empty table. Please specify at least info.features to create an empty table with the right type.)schemar)rYallr<copydeepcopyosr2joinrIrTrBrJfeaturesr Z from_batchespar\r;r)rMfilesrSZ pa_tablesrVZf_dictpa_tablerrr _read_filess"    zBaseReader._read_filescCs t||||j|jd}|j}|S)z?Return list of dict {'filename': str, 'skip': int, 'take': int})r*r+)rFrKrIr")rMr'r)r(r"rdrrrget_file_instructionssz BaseReader.get_file_instructionscCs6||||}|s&d|d}t||j|||dS)a Returns Dataset instance(s). Args: name (str): name of the dataset. instructions (ReadInstruction): instructions to read. Instruction can be string and will then be passed to the Instruction constructor as it. split_infos (list of SplitInfo proto): the available splits for dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a single Dataset instance. z Instruction "z" corresponds to no data!)rdoriginal_instructionsrS)rgr< read_files)rMr'Z instructionsr(rSrdmsgrrrreads  zBaseReader.readN)Nr&r)rdrhcCsF|j||d}|dk r0ddlm}|t|}nd}||j|d}|S)aJReturns single Dataset instance for the set of file instructions. Args: files: List[dict(filename, skip, take)], the files information. The filenames contains the relative path, not absolute. skip/take indicates which example read in the file: `ds.skip().take()` original_instructions: store the original instructions used to build the dataset split in the dataset. in_memory (bool, default False): Whether to copy the data in-memory. Returns: kwargs to build a Dataset instance. rXNr)r)Z arrow_tabler/r3)rfsplitsrr9rJ)rMrdrhrSrerr3Zdataset_kwargsrrrris zBaseReader.read_filesdownload_configc CsXtd|tjd}z\tj|d}t|tjd}t|tj|j d|j dk rp|j |j |j Wn,t k r}zt|dW5d}~XYnXz|j jD]v}|j|j j||j jd}|D]P} tt| d|j } tj|| } t| tjd|d} t| | dqqWn.t k rR}zt|dW5d}~XYnXdS)a% Download the dataset files from the Hf GCS Args: dl_cache_dir: `str`, the local cache directory used to download files relative_data_dir: `str`, the relative directory of the remote files from the `datasets` directory on GCS. /zdataset_info.jsonN)r'r)r(r5rm)HF_GCP_BASE_URLreplacer`sepr2rarshutilmoverIrJupdatefrom_directoryFileNotFoundErrorrrlrgZ builder_namevaluesr9r relative_tor) rMrnZrelative_data_dirZremote_cache_dirZremote_dataset_infoZdownloaded_dataset_infoerrr3r"Zfile_instructionZfile_to_downloadZremote_prepared_filenameZdownloaded_prepared_filenamerrrdownload_from_hf_gcss4     zBaseReader.download_from_hf_gcs)F)F)F)NF)rrrrr9rrNrrTrfrgrkrr%rrir r{rrrrrGs   rGcsPeZdZdZeeddfdd Zd eddd Ze d edd d Z Z S) ArrowReaderz Build a Dataset object out of Instruction instance(s). This Reader uses either memory mapping or file descriptors (in-memory) on arrow files. rrHcst||d|_dS)zInitializes ArrowReader. Args: path (str): path where Arrow files are stored. info (DatasetInfo): info about the dataset. arrowNsuperrNrKrL __class__rrrN?szArrowReader.__init__FrOcCs|dd|kr|dndd|kr(|dnd}}}tj||d}|dkrVt||}|dk r|dk r|dkrz|t|ks|||}|S)rPr5r6Nr7rXrr)r| read_tablerYslice)rMrRrSr5r6r7tablerrrrTIs  $ z$ArrowReader._get_table_from_filenamecCs|rtnt}||S)z Read table from file. Args: filename (str): File name of the table. in_memory (bool, default=False): Whether to copy the data in-memory. Returns: pyarrow.Table )r r from_file)r5rSZ table_clsrrrrXs zArrowReader.read_table)F)F) rrrrr9rrNrrT staticmethodr __classcell__rrrrr|9s  r|cs4eZdZdZeeddfdd ZddZZS) ParquetReaderzv Build a Dataset object out of Instruction instance(s). This Reader uses memory mapping on parquet files. rrHcst||d|_dS)zInitializes ParquetReader. Args: path (str): path where tfrecords are stored. info (DatasetInfo): info about the dataset. parquetNr~rLrrrrNnszParquetReader.__init__cKsv|dd|kr|dndd|kr(|dnd}}}tj|dd}|dk rr|dk rr|dkrf|t|ksr|||}|S)rPr5r6Nr7T)Z memory_mapr)pqrrYr)rMrRkwargsr5r6r7rerrrrTxs $ z&ParquetReader._get_table_from_filename) rrrrr9rrNrTrrrrrrhs rc@s*eZdZUdZeed<eed<eed<dS)_AbsoluteInstructionz?A machine friendly slice: defined absolute positive boundaries.r?r@rAN)rrrrr9r$r#rrrrrs rc@sbeZdZUdZeed<dZeeed<dZ eeed<dZ eeed<dZ eeed<dd Z dS) _RelativeInstructionzHRepresents a single parsed slicing instruction, can use % and negatives.r?Nr@rAunitroundingcCs|jdk r|jdkrtd|jdk r8|jdkr8td|jdkrT|jdk rTtd|jdkr~|jdk r~t|jdkr~td|jdkr|jdk rt|jdkrtd|jdkr|jdkrd n|j|jd <dS) N)%abszunit must be either % or abs)closestZpct1_dropremainderz5rounding must be either closest or pct1_dropremainderrzAIt is forbidden to specify rounding if not using percent slicing.dz2Percent slice boundaries must be > -100 and < 100.rr)rr<rr@rrA__dict__rMrrr __post_init__s""z"_RelativeInstruction.__post_init__) rrrrr9r$r@rr#rArrrrrrrrs rcCst|}|std||ds0|dr4dnd}t|d|d|drbt|dnd |d r|t|d nd |d S) z)Returns ReadInstruction for given string.z!Unrecognized instruction format: Zfrom_pctZto_pctrrr3rfromNrA) split_namerr@rAr) _SUB_SPEC_REmatchr<groupr&r#)specresrrrr_str_to_read_instructions rcCs&|dkrd}t||t|dS)NrzUsing "pct1_dropremainder" rounding on a split with less than 100 elements is forbidden: it always results in an empty dataset.Y@)r<mathtrunc)boundaryr!rjrrr_pct_to_abs_pct1s rcCstt||dS)Nr)r#round)rr!rrr_pct_to_abs_closestsrcCs4|jdkrtnt}|j}||kr:td|dt|d||}|j}|j}|jdkr|dkrddn|||}|dkrz|n|||}n |dkrdn|}|dkr|n|}t ||kst ||krd|pd d |pd d |d }t||dkr||}n|dkrd}|dkr||}n||kr(d}t |||S) zReturns _AbsoluteInstruction instance for given RelativeInstruction. Args: rel_instr: RelativeInstruction instance. name2len: dict {split_name: num_examples}. rzUnknown split "z". Should be one of .rNrzRequested slice [:z] incompatible with z examples.) rrrr?r<listr@rArrr) rel_instrrDZ pct_to_absr3r!r@rArjrrr_rel_to_abs_instrs0       rc@sbeZdZdZddZeddZdddZed d Zd d Z d dZ ddZ ddZ ddZ dS)r&aReading instruction for a dataset. Examples:: # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec('test[:33%]')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction('test', to=33, unit='%')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%')) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%]+train[1:-1]') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%]+train[1:-1]')) ds = datasets.load_dataset('mnist', split=( datasets.ReadInstruction('test', to=33, unit='%') + datasets.ReadInstruction('train', from_=1, to=-1, unit='abs'))) # The following lines are equivalent: ds = datasets.load_dataset('mnist', split='test[:33%](pct1_dropremainder)') ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction.from_spec( 'test[:33%](pct1_dropremainder)')) ds = datasets.load_dataset('mnist', split=datasets.ReadInstruction( 'test', from_=0, to=33, unit='%', rounding="pct1_dropremainder")) # 10-fold validation: tests = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', from_=k, to=k+10, unit='%') for k in range(0, 100, 10)]) trains = datasets.load_dataset( 'mnist', [datasets.ReadInstruction('train', to=k, unit='%') + datasets.ReadInstruction('train', from_=k+10, unit='%') for k in range(0, 100, 10)]) cCs ||_dSrU_relative_instructions)rMrelative_instructionsrrr_initszReadInstruction._initcCs||}|||S)zCReturns ReadInstruction obj initialized with relative_instructions.)__new__r)clsrresultrrr,_read_instruction_from_relative_instructionss  z.rN)r9_ADDITION_SEP_REr3r<rsum)rrsubsr)rrrr=9s   zReadInstruction.from_specc Csg}|jD]}|j}|jdk s(|jdk r|j}|j}|j}|j}|dkrL|nd}|dk rdt||nd}|dk r|t||nd}d|d|d}|dkr|dk r|dkrd|dnd} ||| 7}||q d |S) Nrr[r]r()+) rr?r@rArrr9rBra) rMZrel_instr_specsrZrel_instr_specr@rArrZ slice_strZ rounding_strrrrto_specXs" &  zReadInstruction.to_speccCsjt|tsd}t||j}|j}|djdkr\|djdkr\|jdj|djkr\td|||S)zEReturns a new ReadInstruction obj, result of appending other to self.zAReadInstruction can only be added to another ReadInstruction obj.rrzPIt is forbidden to sum ReadInstruction instances with different rounding values.)r8r&r:rrrr<r)rMotherrjZself_risZ other_risrrr__add__ls   zReadInstruction.__add__cCs|SrU)rrrrr__str__{szReadInstruction.__str__cCsd|jdS)NzReadInstruction(rrrrrr__repr__~szReadInstruction.__repr__csfdd|jDS)aZTranslate instruction into a list of absolute instructions. Those absolute instructions are then to be added together. Args: name2len (`dict`): Associating split names to number of examples. Returns: list of _AbsoluteInstruction instances (corresponds to the + in spec). csg|]}t|qSr)r)r.rrDrrr[sz/ReadInstruction.to_absolute..r)rMrDrrrr>s zReadInstruction.to_absolute)NNNN)rrrrr classmethodrrNr=rrrrr>rrrrr&s&   )NN)?rr^rr`rers dataclassesrpathlibrtypingrrrrZpyarrowrcZpyarrow.parquetrrZdownload.download_configr Znamingr r rr r rrutilsrZutils.file_utilsrr/rrlrrZ get_loggerrloggerrpcompileXrrConnectionErrorrrr r9rFrGr|rrrrrrrr&rrrrsl             H/ "