U
+-e @ s* d dl Z d dlZd dlmZ d dlmZmZmZmZm Z m
Z
d dl Z d dl Z d dl
mZ d dlmZ d dlmZmZmZmZmZmZmZ d dlmZmZm Z dd l m Z dd
l m Z m!Z!m"Z" dd l#m$Z$ ee%j&d d
Z'ee%j&d d Z(e)dZ*e$e+Z,G dd dZ-G dd de-Z.G dd de-Z/G dd de-Z0e1ed dddZ2e
e1ef ee d d d Z3e
e1ef edd d d!Z4dd"ddddd#e1e1e1e1e1ee1e1ee1 e5ee1 ee1 ee1 ee1 ed$d%d&Z6e"dd"ddddd"dd'e1eee1 e5ee1 ee1 ee1 ee1 e5ee1 e1d( d)d*Z7dS )+ N)Path)AnyDictLiteralOptionalTypeUnion)hf_hub_download) upload_file)CardDataDatasetCardData
EvalResult
ModelCardData
SpaceCardDataeval_results_to_model_indexmodel_index_to_eval_results) get_sessionis_jinja_available yaml_dump )
REPOCARD_NAME)EntryNotFoundErrorSoftTemporaryDirectoryvalidate_hf_hub_args)
get_loggerZ templateszmodelcard_template.mdzdatasetcard_template.mdz1^(\s*---[\r\n]+)([\S\s]*?)([\r\n]+---(\r\n|\n|$))c
@ s e Zd ZeZeZdZd ee dddZ
e dd Z e j
edd dZ d
d Zeeef d d
dZed eeef ee ee e dddZd ee dddZd eee ee ee ee ee ee ee dddZed eee dddZdS )!RepoCardmodelF)contentignore_metadata_errorsc C s || _ || _dS )a Initialize a RepoCard from string content. The content should be a
Markdown file with a YAML block at the beginning and a Markdown body.
Args:
content (`str`): The content of the Markdown file.
Example:
```python
>>> from huggingface_hub.repocard import RepoCard
>>> text = '''
... ---
... language: en
... license: mit
... ---
...
... # My repo
... '''
>>> card = RepoCard(text)
>>> card.data.to_dict()
{'language': 'en', 'license': 'mit'}
>>> card.text
'\n# My repo\n'
```
Raises the following error:
- [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
when the content of the repo card metadata is not a dictionary.
N)r r )selfr r r Y/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/huggingface_hub/repocard.py__init__* s $zRepoCard.__init__c C s6 t | jp d}d| | jj|d | d| | j S )zLThe content of the RepoCard, including the YAML block and the Markdown body.
---)
line_break)_detect_line_ending_contentdatato_yamltext)r r% r r r! r Q s zRepoCard.contentr c C s || _ t|}|rZ|d}|| d | _t|}|dkrFi }t|t snt
dnt d i }|| _| j
f |d| ji| _dS )z Set the content of the RepoCard. N)repo card metadata block should be a dictzBRepo card metadata block was not found. Setting CardData to empty.r )r' REGEX_YAML_BLOCKsearchgroupendr* yaml safe_load
isinstancedict
ValueErrorloggerwarningcard_data_classr r( )r r match
yaml_blockZ data_dictr r r! r W s
c C s | j S )Nr+ )r r r r! __str__q s zRepoCard.__str__)filepathc C sF t |}|jjddd t|dddd}|t| W 5 Q R X dS )a{ Save a RepoCard to a file.
Args:
filepath (`Union[Path, str]`): Filepath to the markdown file to save.
Example:
```python
>>> from huggingface_hub.repocard import RepoCard
>>> card = RepoCard("---\nlanguage: en\n---\n# This is a test repo card")
>>> card.save("/tmp/test.md")
```
T)parentsexist_okw utf-8modenewlineencodingN)r parentmkdiropenwritestr)r r= fr r r! savet s z
RepoCard.saveN)repo_id_or_path repo_typetokenr c
C s t | rt |}n6t|tr>> from huggingface_hub.repocard import RepoCard
>>> card = RepoCard.load("nateraw/food")
>>> assert card.data.tags == ["generated_from_trainer", "image-classification", "pytorch"]
```
)rO rP z.Cannot load RepoCard: path not found on disk (z).rrA rB rC )r N)
r existsr4 rK r r rO r6 rI read)clsrN rO rP r Z card_pathrL r r r! load s $
z
RepoCard.loadrO c
C s |p| j }|t| d}ddi}z t jd||d}| W n@ tjjk
r~ } z |jdkrjt |j
n|W 5 d}~X Y nX dS )a Validates card against Hugging Face Hub's card validation logic.
Using this function requires access to the internet, so it is only called
internally by [`huggingface_hub.repocard.RepoCard.push_to_hub`].
Args:
repo_type (`str`, *optional*, defaults to "model"):
The type of Hugging Face repo to push to. Options are "model", "dataset", and "space".
If this function is called from a child class, the default will be the child class's `repo_type`.
Raises the following errors:
- [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
if the card fails validation checks.
- [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError)
if the request to the Hub API fails for any other reason.
)ZrepoTyper Acceptz
text/plainz(https://huggingface.co/api/validate-yaml)headersi N) rO rK r postraise_for_statusrequests
exceptions HTTPError status_coder6 r* )r rO bodyrX rQ excr r r! validate s
zRepoCard.validate)repo_idrP rO commit_messagecommit_descriptionrevision create_pr
parent_commitc
C sf |p| j }| j|d t @} t| t }
|
t| tt|
t||||||||d
} W 5 Q R X | S )aA Push a RepoCard to a Hugging Face Hub repo.
Args:
repo_id (`str`):
The repo ID of the Hugging Face Hub repo to push to. Example: "nateraw/food".
token (`str`, *optional*):
Authentication token, obtained with `huggingface_hub.HfApi.login` method. Will default to
the stored token.
repo_type (`str`, *optional*, defaults to "model"):
The type of Hugging Face repo to push to. Options are "model", "dataset", and "space". If this
function is called by a child class, it will default to the child class's `repo_type`.
commit_message (`str`, *optional*):
The summary / title / first line of the generated commit.
commit_description (`str`, *optional*)
The description of the generated commit.
revision (`str`, *optional*):
The git revision to commit from. Defaults to the head of the `"main"` branch.
create_pr (`bool`, *optional*):
Whether or not to create a Pull Request with this commit. Defaults to `False`.
parent_commit (`str`, *optional*):
The OID / SHA of the parent commit, as a hexadecimal string. Shorthands (7 first characters) are also supported.
If specified and `create_pr` is `False`, the commit will fail if `revision` does not point to `parent_commit`.
If specified and `create_pr` is `True`, the pull request will be created from `parent_commit`.
Specifying `parent_commit` ensures the repo has not changed before committing the changes, and can be
especially useful if the repo is updated / committed to concurrently.
Returns:
`str`: URL of the commit which updated the card metadata.
rV )
Zpath_or_fileobjZ path_in_reporb rP rO rc rd rf re rg )rO ra r r r
write_textrK r
) r rb rP rO rc rd re rf rg ZtmpdirZtmp_pathurlr r r! push_to_hub s$ )
zRepoCard.push_to_hub card_data
template_pathc K sf t rddl}ntd| }|| |t|p<| j }|j
f d| i|}| |S )a' Initialize a RepoCard from a template. By default, it uses the default template.
Templates are Jinja2 templates that can be customized by passing keyword arguments.
Args:
card_data (`huggingface_hub.CardData`):
A huggingface_hub.CardData instance containing the metadata you want to include in the YAML
header of the repo card on the Hugging Face Hub.
template_path (`str`, *optional*):
A path to a markdown file with optional Jinja template variables that can be filled
in with `template_kwargs`. Defaults to the default template.
Returns:
[`huggingface_hub.repocard.RepoCard`]: A RepoCard instance with the specified card data and content from the
template.
r NzjUsing RepoCard.from_template requires Jinja2 to be installed. Please install it with `pip install Jinja2`.rl ) r jinja2 ImportErrorto_dictcopyupdateTemplater default_template_path read_textrenderr) )rT rl rm template_kwargsrn kwargstemplater r r r!
from_template$ s
zRepoCard.from_template)F)NNF)N)NNNNNNN)N)__name__
__module__ __qualname__r r9 TEMPLATE_MODELCARD_PATHrt rO rK boolr" propertyr setterr< r r rM classmethodr rU ra rj rz r r r r! r % sX '
5* ? r c s: e Zd ZeZeZdZedee e
d fdd
Z Z S ) ModelCardr Nrk c s t j||f|S )a
Initialize a ModelCard from a template. By default, it uses the default template, which can be found here:
https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md
Templates are Jinja2 templates that can be customized by passing keyword arguments.
Args:
card_data (`huggingface_hub.ModelCardData`):
A huggingface_hub.ModelCardData instance containing the metadata you want to include in the YAML
header of the model card on the Hugging Face Hub.
template_path (`str`, *optional*):
A path to a markdown file with optional Jinja template variables that can be filled
in with `template_kwargs`. Defaults to the default template.
Returns:
[`huggingface_hub.ModelCard`]: A ModelCard instance with the specified card data and content from the
template.
Example:
```python
>>> from huggingface_hub import ModelCard, ModelCardData, EvalResult
>>> # Using the Default Template
>>> card_data = ModelCardData(
... language='en',
... license='mit',
... library_name='timm',
... tags=['image-classification', 'resnet'],
... datasets=['beans'],
... metrics=['accuracy'],
... )
>>> card = ModelCard.from_template(
... card_data,
... model_description='This model does x + y...'
... )
>>> # Including Evaluation Results
>>> card_data = ModelCardData(
... language='en',
... tags=['image-classification', 'resnet'],
... eval_results=[
... EvalResult(
... task_type='image-classification',
... dataset_type='beans',
... dataset_name='Beans',
... metric_type='accuracy',
... metric_value=0.9,
... ),
... ],
... model_name='my-cool-model',
... )
>>> card = ModelCard.from_template(card_data)
>>> # Using a Custom Template
>>> card_data = ModelCardData(
... language='en',
... tags=['image-classification', 'resnet']
... )
>>> card = ModelCard.from_template(
... card_data=card_data,
... template_path='./src/huggingface_hub/templates/modelcard_template.md',
... custom_template_var='custom value', # will be replaced in template if it exists
... )
```
superrz rT rl rm rw __class__r r! rz O s HzModelCard.from_template)N)
r{ r| r} r r9 r~ rt rO r r rK rz
__classcell__r r r r! r J s r c s: e Zd ZeZeZdZedee e
d fdd
Z Z S ) DatasetCarddatasetNrk c s t j||f|S )a Initialize a DatasetCard from a template. By default, it uses the default template, which can be found here:
https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/datasetcard_template.md
Templates are Jinja2 templates that can be customized by passing keyword arguments.
Args:
card_data (`huggingface_hub.DatasetCardData`):
A huggingface_hub.DatasetCardData instance containing the metadata you want to include in the YAML
header of the dataset card on the Hugging Face Hub.
template_path (`str`, *optional*):
A path to a markdown file with optional Jinja template variables that can be filled
in with `template_kwargs`. Defaults to the default template.
Returns:
[`huggingface_hub.DatasetCard`]: A DatasetCard instance with the specified card data and content from the
template.
Example:
```python
>>> from huggingface_hub import DatasetCard, DatasetCardData
>>> # Using the Default Template
>>> card_data = DatasetCardData(
... language='en',
... license='mit',
... annotations_creators='crowdsourced',
... task_categories=['text-classification'],
... task_ids=['sentiment-classification', 'text-scoring'],
... multilinguality='monolingual',
... pretty_name='My Text Classification Dataset',
... )
>>> card = DatasetCard.from_template(
... card_data,
... pretty_name=card_data.pretty_name,
... )
>>> # Using a Custom Template
>>> card_data = DatasetCardData(
... language='en',
... license='mit',
... )
>>> card = DatasetCard.from_template(
... card_data=card_data,
... template_path='./src/huggingface_hub/templates/datasetcard_template.md',
... custom_template_var='custom value', # will be replaced in template if it exists
... )
```
r r r r r! rz s 8zDatasetCard.from_template)N)
r{ r| r} r r9 TEMPLATE_DATASETCARD_PATHrt rO r r rK rz r r r r r! r s r c @ s e Zd ZeZeZdZdS ) SpaceCardspaceN)r{ r| r} r r9 r~ rt rO r r r r! r s r )
r#
N)r returnc C sV | d}| d}| d}|| dkr.dS ||krB||krBdS ||krNdS dS dS )zDetect the line ending of a string. Used by RepoCard to avoid making huge diff on newlines.
Uses same implementation as in Hub server, keep it in sync.
Returns:
str: The detected line ending of the string.
r r# r r N)count)r crlfZcrlfr r r! r& s
r& )
local_pathr c C sV t | }t|}|rN|d}t|}|d ks@t|trD|S t dnd S d S )Nr, r- )
r ru r. r/ r0 r2 r3 r4 r5 r6 )r r r: r; r( r r r!
metadata_load s
r )r r( r c C s d}d}t j| rbt| dddd8}| }t|jtrF|jd }nt|jtrX|j}W 5 Q R X t| dddd}t |d|d }t
|}|r|d
| d | | d | ||
d
}nd | | d | | }|| | W 5 Q R X d
S ) a&
Save the metadata dict in the upper YAML part Trying to preserve newlines as
in the existing file. Docs about open() with newline="" parameter:
https://docs.python.org/3/library/functions.html?highlight=open#open Does
not work with "^M" linebreaks, which are replaced by
r# rA rQ utf8)rE rF r r@ F) sort_keysr% Nr$ )ospathrR rI rS r4 newlinestuplerK r r. r/ startr1 rJ close)r r( r% r ZreadmeZ data_yamlr: outputr r r!
metadata_save s"
6
r F)metrics_configmetrics_verifieddataset_config
dataset_splitdataset_revisionmetrics_verification_token)model_pretty_nametask_pretty_nametask_idmetrics_pretty_name
metrics_id
metrics_valuedataset_pretty_name
dataset_idr r r r r r r c C s0 dt | t||||||||| |
|
| | d
gdiS )u
Creates a metadata dict with the result from a model evaluated on a dataset.
Args:
model_pretty_name (`str`):
The name of the model in natural language.
task_pretty_name (`str`):
The name of a task in natural language.
task_id (`str`):
Example: automatic-speech-recognition. A task id.
metrics_pretty_name (`str`):
A name for the metric in natural language. Example: Test WER.
metrics_id (`str`):
Example: wer. A metric id from https://hf.co/metrics.
metrics_value (`Any`):
The value from the metric. Example: 20.0 or "20.0 ± 1.2".
dataset_pretty_name (`str`):
The name of the dataset in natural language.
dataset_id (`str`):
Example: common_voice. A dataset id from https://hf.co/datasets.
metrics_config (`str`, *optional*):
The name of the metric configuration used in `load_metric()`.
Example: bleurt-large-512 in `load_metric("bleurt", "bleurt-large-512")`.
metrics_verified (`bool`, *optional*, defaults to `False`):
Indicates whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. Automatically computed by Hugging Face, do not set.
dataset_config (`str`, *optional*):
Example: fr. The name of the dataset configuration used in `load_dataset()`.
dataset_split (`str`, *optional*):
Example: test. The name of the dataset split used in `load_dataset()`.
dataset_revision (`str`, *optional*):
Example: 5503434ddd753f426f4b38109466949a1217c2bb. The name of the dataset dataset revision
used in `load_dataset()`.
metrics_verification_token (`bool`, *optional*):
A JSON Web Token that is used to verify whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not.
Returns:
`dict`: a metadata dict with the result from a model evaluated on a dataset.
Example:
```python
>>> from huggingface_hub import metadata_eval_result
>>> results = metadata_eval_result(
... model_pretty_name="RoBERTa fine-tuned on ReactionGIF",
... task_pretty_name="Text Classification",
... task_id="text-classification",
... metrics_pretty_name="Accuracy",
... metrics_id="accuracy",
... metrics_value=0.2662102282047272,
... dataset_pretty_name="ReactionJPEG",
... dataset_id="julien-c/reactionjpeg",
... dataset_config="default",
... dataset_split="test",
... )
>>> results == {
... 'model-index': [
... {
... 'name': 'RoBERTa fine-tuned on ReactionGIF',
... 'results': [
... {
... 'task': {
... 'type': 'text-classification',
... 'name': 'Text Classification'
... },
... 'dataset': {
... 'name': 'ReactionJPEG',
... 'type': 'julien-c/reactionjpeg',
... 'config': 'default',
... 'split': 'test'
... },
... 'metrics': [
... {
... 'type': 'accuracy',
... 'value': 0.2662102282047272,
... 'name': 'Accuracy',
... 'verified': False
... }
... ]
... }
... ]
... }
... ]
... }
True
```
model-index)
Z task_nameZ task_type metric_name metric_type metric_valueZ dataset_nameZ dataset_typeZ
metric_configverified verify_tokenr r r )
model_name eval_results)r r
)r r r r r r r r r r r r r r r r r! metadata_eval_result" s* i r )rO overwriterP rc rd re rf rg ) rb metadatarO r rP rc rd re rf rg r c
C s |dk r |nd}|dks |dkr&t }
n*|dkr4t}
n |dkrBt}
ntd| z|
j| ||d} W n0 tk
r |dkrtd|
t } Y nX | D ]2\} }
| d krd
|
d krt | d | |
d d
< t
|
\}}| j j dkr|| j _ || j _
n| j j }|D ]}d
}|D ]\}||r||krN|sNtd|j d|j dd}|j|_|jdkr|j|_q|s| j j | qq| j | dk r|s| j | |
krtd| dq|
| j | < q| j| ||||||| dS )a
Updates the metadata in the README.md of a repository on the Hugging Face Hub.
If the README.md file doesn't exist yet, a new one is created with metadata and an
the default ModelCard or DatasetCard template. For `space` repo, an error is thrown
as a Space cannot exist without a `README.md` file.
Args:
repo_id (`str`):
The name of the repository.
metadata (`dict`):
A dictionary containing the metadata to be updated.
repo_type (`str`, *optional*):
Set to `"dataset"` or `"space"` if updating to a dataset or space,
`None` or `"model"` if updating to a model. Default is `None`.
overwrite (`bool`, *optional*, defaults to `False`):
If set to `True` an existing field can be overwritten, otherwise
attempting to overwrite an existing field will cause an error.
token (`str`, *optional*):
The Hugging Face authentication token.
commit_message (`str`, *optional*):
The summary / title / first line of the generated commit. Defaults to
`f"Update metadata with huggingface_hub"`
commit_description (`str` *optional*)
The description of the generated commit
revision (`str`, *optional*):
The git revision to commit from. Defaults to the head of the
`"main"` branch.
create_pr (`boolean`, *optional*):
Whether or not to create a Pull Request from `revision` with that commit.
Defaults to `False`.
parent_commit (`str`, *optional*):
The OID / SHA of the parent commit, as a hexadecimal string. Shorthands (7 first characters) are also supported.
If specified and `create_pr` is `False`, the commit will fail if `revision` does not point to `parent_commit`.
If specified and `create_pr` is `True`, the pull request will be created from `parent_commit`.
Specifying `parent_commit` ensures the repo has not changed before committing the changes, and can be
especially useful if the repo is updated / committed to concurrently.
Returns:
`str`: URL of the commit which updated the card metadata.
Example:
```python
>>> from huggingface_hub import metadata_update
>>> metadata = {'model-index': [{'name': 'RoBERTa fine-tuned on ReactionGIF',
... 'results': [{'dataset': {'name': 'ReactionGIF',
... 'type': 'julien-c/reactiongif'},
... 'metrics': [{'name': 'Recall',
... 'type': 'recall',
... 'value': 0.7762102282047272}],
... 'task': {'name': 'Text Classification',
... 'type': 'text-classification'}}]}]}
>>> url = metadata_update("hf-internal-testing/reactiongif-roberta-card", metadata)
```
Nz$Update metadata with huggingface_hubr r r zUnknown repo_type: )rP rO zJCannot update metadata on a Space that doesn't contain a `README.md` file.r namer r Fz6You passed a new value for the existing metric 'name: z, type: z6'. Set `overwrite=True` to overwrite existing metrics.Tz9You passed a new value for the existing meta data field 'z7'. Set `overwrite=True` to overwrite existing metadata.)rP rO rc rd rf re rg )r r r r6 rU r rz r itemsgetattrr r( r r Zis_equal_except_valuer r r r r appendgetrj )rb r rO r rP rc rd re rf rg Z
card_classZcardkeyvaluer Z new_resultsZexisting_resultsZ
new_resultZ result_foundZexisting_resultr r r! metadata_update sh D
*
r )8r repathlibr typingr r r r r r r[ r2 Z huggingface_hub.file_downloadr Zhuggingface_hub.hf_apir
Z huggingface_hub.repocard_datar r r
r r r r Zhuggingface_hub.utilsr r r constantsr utilsr r r Z
utils.loggingr __file__rG r~ r compiler. r{ r7 r r r r rK r& r r r r r r r r r! s $
'P@
*