U -eS1@s$ddlmZddlZddlmZddlmZmZm Z ddl Z ddl mZddl m Z dZedeZeeZee eefd d d Zed \ZZd ddgZe eeeeefZGdddZe dGdddeZedkr ddlmZeddZe de!Z"ee"j#Z#e$e#Z%dS)N)Path)AnyListTuple) resources) deprecatedzDhttps://github.com/huggingface/datasets/tree/main/src/datasets/utils/)resourcereturncCs$tt|}t|td|fS)Nz /resources/) pkg_resources read_textryamlZ safe_load BASE_REF_URL)r contentrV/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/datasets/utils/readme.pyload_yaml_resources rzreadme_structure.yamlz[Needs More Information]z[More Information Needed]zj(https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)c@sTeZdZdeeeeedddZdedddZee d d d Z ed d dZ dS)SectionNF)namelevellinessuppress_parsing_errorscCsJ||_||_||_d|_d|_i|_g|_g|_|jdk rF|j|ddS)NTr) rrrtext is_empty_textrparsing_error_listparsing_warning_listparse)selfrrrrrrr__init__$s zSection.__init__rcCsd}g}d}|jD]}|ddkr(qq|ddddkrF| }q|d|jdkr|s|dkrt||jd||j|<g}n>|gkr|jd|7_|jdkr|jtkrd|_ g}d|d dd}q| |q|dkr2||jkr|j d |d t||jd||j|<n@|gkrr|jd|7_|jdkrr|jtkrrd|_ |jdkr|s|j gks|j gkrd d d|j |j D}}d|j d|}t|dS)NrF z```r# rz)Multiple sections with the same heading `z:` have been found. Please keep only one of these sections. css|]}d|VqdS)- Nr.0xrrr Vsz Section.parse..z=The following issues were found while parsing the README at ``: )rstripsplitrrrrjoin FILLER_TEXTrappendrrr ValueError)r rZcurrent_sub_levelZ current_linesZ code_startlineerrors error_stringrrrr0sD       z Section.parse) structurer c Csg}g}|ddkr8|jr8|jikr8|d|jd|ddkr^|jr^|d|jd|dd k r~|jikrd d |dD}|d |jd ddd |Ddndd |dD}d}t|D]r\}}||jkr|d |jd|dd}q|jdkrqn|j||d|\} } } || 7}|| 7}q|r~|jD]*}||krR|d|jd|dqR|ri||fS|||fSd S)aCValidates a Section class object recursively using the structure provided as a dictionary. Args: structute (:obj: `dict`): The dictionary representing expected structure. Returns: :obj: `ReadmeValidatorOutput`: The dictionary representation of the section, and the errors. Z allow_emptyFz"Expected some content in section `z` but it is empty.Zallow_empty_textzExpected some text in section `z4` but it is empty (text in subsections are ignored). subsectionsNcSsg|] }|dqSrrr)Z subsectionrrr xsz$Section.validate..z Section `z&` expected the following subsections: , cSsg|]}d|dqS`rr(rrrr:{sz. Found 'None'.cSsg|] }|dqSr8rr9rrrr:sz` is missing subsection: `z`.Tz###r=z` has an extra subsection: `z[`. Skipping further validation checks for this subsection as expected structure is unknown.) rrr1rr/ enumeratervalidateto_dict) r r6 error_list warning_listvaluesZstructure_namesZhas_missing_subsectionsidxr_Zsubsec_error_listZsubsec_warning_listrrrr?ZsL     "        zSection.validate)r cCs$|j|j|jdd|jDdS)z3Returns the dictionary representation of a section.cSsg|] }|qSr)r@)r)valuerrrr:sz#Section.to_dict..)rrrr7)rrrrrCr rrrr@s zSection.to_dict)NF)F) __name__ __module__ __qualname__strrboolr!rdictReadmeValidatorOutputr?r@rrrrr#s *Grz*Use `huggingface_hub.DatasetCard` instead.cseZdZdeeeeedfdd ZddZe de eedd d Z e deeeed d dZ dedfdd Z ddZddZZS)ReadMeNF)rrr6rcsBtj|dd||_d|_d|_||_|jdk r>|j|ddS)Nr)rrrr)superr!r6yaml_tags_line_count tag_countrr)r rrr6r __class__rrr!s zReadMe.__init__cCst|jdkr|t\}}}n||j\}}}|gks>|gkrpddd||D}d|jd|}t|dS)Nr&cSsg|] }d|qS)r'rr(rrrr:sz#ReadMe.validate..z3The following issues were found for the README at `r,)r6 _validatereadme_structurer/rr2)r rrArBr4r5rrrr?s zReadMe.validate)pathr6rc Cs0t|dd}|}W5QRX|||||dS)Nzutf-8)encodingr)open readlines)clsrXr6rfrrrr from_readmeszReadMe.from_readmeroot)stringr6 root_namercCs|d}|||||dS)Nr&r)r.)r\r`r6rarrrrr from_strings zReadMe.from_stringrcsd}|jD]D}|jd7_|ddkrF|jd7_|jdkrFqP|d7}q |jdkrp|j|dd|_n|j|jd|_tj|ddS)Nrrr"z---r)rrRr-rSrQr)r r line_countr3rTrrrs    z ReadMe.parsecCs t|S)z>Returns the string of dictionary representation of the ReadMe.)rKr@rGrrr__str__szReadMe.__str__c Csg}g}|jdkr|dn*|jdkr4|dn|jdkrH|dt|j}|dkr|dddd t|jDd nj|dkr|d nVt|jd}|d r|j| |d d\}}}||7}||7}n |d|ri||fS| ||fSdS)Nrz-Empty YAML markers are present in the README.z*No YAML markers are present in the README.rz2Only the start of YAML tags present in the README.z-The README has several first-level headings: r;cSsg|]}d|dqSr<rr(rrrr:sz$ReadMe._validate..zL. Only one heading is expected. Skipping further validation for this README.zmThe README has no first-level headings. One heading is expected. Skipping further validation for this README.zDataset Card forr7zuNo first-level heading starting with `Dataset Card for` found in README. Skipping further validation for this README.) rRr1rSlenrkeysr/list startswithr?r@) r rWrArBZnum_first_level_keysZ start_keyrEZsec_error_listZsec_warning_listrrrrVs<      $     zReadMe._validate)NF)NF)Nr_F)F)rHrIrJrKrrMrLr!r? classmethodrr^rbrrerV __classcell__rrrTrrOs"  rO__main__)ArgumentParserz?Validate the content (excluding YAML tags) of a README.md file.)usagereadme_filepath)&importlib.resourcesrr loggingpathlibrtypingrrrrrZdeprecation_utilsrr__file__Zthis_url getLoggerrHloggerrKrrWZknown_readme_structure_urlr0rMrNrrOargparsermZap add_argument parse_argsargsror^Zreadmerrrrs6       b