U -e@s$ddlZddlZddlmZddlmZddlmZddlm Z e e Z Gdd d ej Z Gd d d eZGd d d eZGdddeZGdddeZd&eeedddZGdddeZGdddeZGdddeZGdddeZeeedddZd'eeed!d"d#Zd$d%ZdS)(N)sha256)Optional)config) get_loggerc@seZdZdZdZdZdZdS)VerificationModea`Enum` that specifies which verification checks to run. The default mode is `BASIC_CHECKS`, which will perform only rudimentary checks to avoid slowdowns when generating/downloading a dataset for the first time. The verification modes: | | Verification checks | |---------------------------|------------------------------------------------------------------------------ | | `ALL_CHECKS` | Split checks, uniqueness of the keys yielded in case of the GeneratorBuilder | | | and the validity (number of files, checksums, etc.) of downloaded files | | `BASIC_CHECKS` (default) | Same as `ALL_CHECKS` but without checking downloaded files | | `NO_CHECKS` | None | Z all_checksZ basic_checksZ no_checksN)__name__ __module__ __qualname____doc__Z ALL_CHECKSZ BASIC_CHECKSZ NO_CHECKSr r Z/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/utils/info_utils.pyr src@seZdZdZdS)ChecksumVerificationExceptionz>Exceptions during checksums verifications of downloaded files.Nr r r r r r r rr#src@seZdZdZdS)UnexpectedDownloadedFilez(Some downloaded files were not expected.Nrr r r rr'src@seZdZdZdS)ExpectedMoreDownloadedFilesz7Some files were supposed to be downloaded but were not.Nrr r r rr+src@seZdZdZdS)NonMatchingChecksumErrorz?The downloaded file checksum don't match the expected checksum.Nrr r r rr/srexpected_checksumsrecorded_checksumscsdkrtddStttdkrFtttttttdkrvttttfddD}|dk rd|nd}t|dkrtd|d|d td |dS) NzUnable to verify checksums.rcs g|]}||kr|qSr r ).0urlrr r ;sz$verify_checksums..z for zChecksums didn't matchz: zY Set `verification_mode='no_checks'` to skip checksums verification and ignore this errorz&All the checksums matched successfully)loggerinfolensetrstrrr)rrZverification_nameZbad_urlsZfor_verification_namer rrverify_checksums3s  r c@seZdZdZdS)SplitsVerificationExceptionz%Exceptions during splis verificationsNrr r r rr!Fsr!c@seZdZdZdS)UnexpectedSplitsz6The expected splits of the downloaded file is missing.Nrr r r rr"Jsr"c@seZdZdZdS)ExpectedMoreSplitsz!Some recorded splits are missing.Nrr r r rr#Nsr#c@seZdZdZdS)NonMatchingSplitsSizesErrorz7The splits sizes don't match the expected splits sizes.Nrr r r rr$Rsr$expected_splitsrecorded_splitscsdkrtddStttdkrFtttttttdkrvttttfddD}t|dkrtt|tddS)NzUnable to verify splits sizes.rcs2g|]*}|j|jkr||dqS))expectedZrecorded)Z num_examples)rnamer%r rr^sz!verify_splits..z$All the splits matched successfully.)rrrrr#rr"r$)r&r'Z bad_splitsr r%r verify_splitsVs    r*T)pathrecord_checksumreturnc sb|rLt}t|d0tfdddD]}||q(|}W5QRXnd}tj||dS)z7Compute the file size and the sha256 checksum of a filerbcs dS)Ni)readr fr rmz(get_size_checksum_dict..r3N) num_byteschecksum)ropeniterupdate hexdigestosr+getsize)r+r,mchunkr5r r0rget_size_checksum_dicths  r>cCs|rtjr|tjkSdSdS)zCheck if `dataset_size` is smaller than `config.IN_MEMORY_MAX_SIZE`. Args: dataset_size (int): Dataset size in bytes. Returns: bool: Whether `dataset_size` is smaller than `config.IN_MEMORY_MAX_SIZE`. FN)rZIN_MEMORY_MAX_SIZE)Z dataset_sizer r ris_small_datasetus  r?)N)T)enumr:hashlibrtypingrrrloggingrr rEnumr Exceptionrrrrdictr r!r"r#r$r*rboolr>r?r r r rs&