U
+-ea. @ s d dl Z d dlmZ d dlmZmZmZmZmZ d dl m
Z d dl m
Z
ddlmZmZmZmZ ddlmZmZmZ dd lmZ dd
lmZmZm Z ddlm
Z e e Z e dddddd ddddd d
d
dd
ddddde!ee! ee! ee! ee!edf ee!edf ee"ed f ee! ee! eeee!f ee e#e"e"eee"e!f e"eeee! e!f eeee! e!f e$ee e!dddZ%dS ) N)Path)DictListLiteralOptionalUnion)tqdm)
thread_map )DEFAULT_REVISIONHF_HUB_ENABLE_HF_TRANSFERHUGGINGFACE_HUB_CACHE
REPO_TYPES)REGEX_COMMIT_HASHhf_hub_downloadrepo_folder_name)HfApi)filter_repo_objectsloggingvalidate_hf_hub_argsauto
F ) repo_typerevisionendpoint cache_dir local_dirlocal_dir_use_symlinks library_namelibrary_version
user_agentproxies etag_timeoutresume_downloadforce_downloadtokenlocal_files_onlyallow_patternsignore_patterns max_workers
tqdm_class)repo_idr r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ returnc s dkr t |dkrt}t tr*t dkr6d tkrVtd dtt tj t
d}|rt |r|n,tj |d|}t
|}| W 5 Q R X tj |d}tj|r|S tdtd }|j
|
d
}|jdk s
td ttd d
|jD ||d}|jtj |d}|krtj |d|}tjtj|dd t
|d}| W 5 Q R X td
fdd }tr|D ]} || qn$t||dt | d||pt d dk rttj S |S )a Download repo files.
Download a whole snapshot of a repo's files at the specified revision. This is useful when you want all files from
a repo, because you don't know which ones you will need a priori. All files are nested inside a folder in order
to keep their actual filename relative to that folder. You can also filter which files to download using
`allow_patterns` and `ignore_patterns`.
If `local_dir` is provided, the file structure from the repo will be replicated in this location. You can configure
how you want to move those files:
- If `local_dir_use_symlinks="auto"` (default), files are downloaded and stored in the cache directory as blob
files. Small files (<5MB) are duplicated in `local_dir` while a symlink is created for bigger files. The goal
is to be able to manually edit and save small files without corrupting the cache while saving disk space for
binary files. The 5MB threshold can be configured with the `HF_HUB_LOCAL_DIR_AUTO_SYMLINK_THRESHOLD`
environment variable.
- If `local_dir_use_symlinks=True`, files are downloaded, stored in the cache directory and symlinked in `local_dir`.
This is optimal in term of disk usage but files must not be manually edited.
- If `local_dir_use_symlinks=False` and the blob files exist in the cache directory, they are duplicated in the
local dir. This means disk usage is not optimized.
- Finally, if `local_dir_use_symlinks=False` and the blob files do not exist in the cache directory, then the
files are downloaded and directly placed under `local_dir`. This means if you need to download them again later,
they will be re-downloaded entirely.
An alternative would be to clone the repo but this requires git and git-lfs to be installed and properly
configured. It is also not possible to filter which files to download when cloning a repository using git.
Args:
repo_id (`str`):
A user or an organization name and a repo name separated by a `/`.
repo_type (`str`, *optional*):
Set to `"dataset"` or `"space"` if downloading from a dataset or space,
`None` or `"model"` if downloading from a model. Default is `None`.
revision (`str`, *optional*):
An optional Git revision id which can be a branch name, a tag, or a
commit hash.
endpoint (`str`, *optional*):
Hugging Face Hub base url. Will default to https://huggingface.co/. Otherwise, one can set the `HF_ENDPOINT`
environment variable.
cache_dir (`str`, `Path`, *optional*):
Path to the folder where cached files are stored.
local_dir (`str` or `Path`, *optional*):
If provided, the downloaded files will be placed under this directory, either as symlinks (default) or
regular files (see description for more details).
local_dir_use_symlinks (`"auto"` or `bool`, defaults to `"auto"`):
To be used with `local_dir`. If set to "auto", the cache directory will be used and the file will be either
duplicated or symlinked to the local directory depending on its size. It set to `True`, a symlink will be
created, no matter the file size. If set to `False`, the file will either be duplicated from cache (if
already exists) or downloaded from the Hub and not cached. See description for more details.
library_name (`str`, *optional*):
The name of the library to which the object corresponds.
library_version (`str`, *optional*):
The version of the library.
user_agent (`str`, `dict`, *optional*):
The user-agent info in the form of a dictionary or a string.
proxies (`dict`, *optional*):
Dictionary mapping protocol to the URL of the proxy passed to
`requests.request`.
etag_timeout (`float`, *optional*, defaults to `10`):
When fetching ETag, how many seconds to wait for the server to send
data before giving up which is passed to `requests.request`.
resume_download (`bool`, *optional*, defaults to `False):
If `True`, resume a previously interrupted download.
force_download (`bool`, *optional*, defaults to `False`):
Whether the file should be downloaded even if it already exists in the local cache.
token (`str`, `bool`, *optional*):
A token to be used for the download.
- If `True`, the token is read from the HuggingFace config
folder.
- If a string, it's used as the authentication token.
local_files_only (`bool`, *optional*, defaults to `False`):
If `True`, avoid downloading the file and return the path to the
local cached file if it exists.
allow_patterns (`List[str]` or `str`, *optional*):
If provided, only files matching at least one pattern are downloaded.
ignore_patterns (`List[str]` or `str`, *optional*):
If provided, files matching any of the patterns are not downloaded.
max_workers (`int`, *optional*):
Number of concurrent threads to download files (1 thread = 1 file download).
Defaults to 8.
tqdm_class (`tqdm`, *optional*):
If provided, overwrites the default behavior for the progress bar. Passed
argument must inherit from `tqdm.auto.tqdm` or at least mimic its behavior.
Note that the `tqdm_class` is not passed to each individual download.
Defaults to the custom HF progress bar that can be disabled by setting
`HF_HUB_DISABLE_PROGRESS_BARS` environment variable.
Returns:
Local folder path (string) of repo snapshot
Raises the following errors:
- [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError)
if `token=True` and the token cannot be found.
- [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if
ETag cannot be determined.
- [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
if some parameter value is invalid
NmodelzInvalid repo type: z. Accepted repo types are: )r, r refsZ snapshotszCannot find an appropriate cached snapshot folder for the specified revision on the local disk and outgoing traffic has been disabled. To enable repo look-ups and downloads online, set 'local_files_only' to False.)r r r! r )r, r r r& z8Repo info returned from server must have a revision sha.c S s g | ]
}|j qS )Z rfilename).0fr0 r0 c/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/huggingface_hub/_snapshot_download.py
s z%snapshot_download..)itemsr( r) T)exist_okwZ repo_filec s( t
|
dS )N)filenamer r r r r r r r r! r" r# r$ r% r& )r r8 r Z commit_hashr r# r% r r r r r" r, r r$ r& r! r0 r3 _inner_hf_hub_download s$ z1snapshot_download.._inner_hf_hub_downloadz Fetching z files)descr* r+ ) r
r
isinstancer strr
ValueErrorospathjoinr r matchopenreadexistsr repo_infoshaAssertionErrorlistr Zsiblingsmakedirsdirnamewriter r lenhf_tqdmrealpath) r, r r r r r r r r r! r" r# r$ r% r& r' r( r) r* r+ Zstorage_folderZref_pathr2 Zsnapshot_folderapirG Zfiltered_repo_filesr; filer0 r: r3 snapshot_download sl }
.
rS )&r@ pathlibr typingr r r r r Z tqdm.autor Z base_tqdmZtqdm.contrib.concurrentr constantsr r r
r Z
file_downloadr r r Zhf_apir utilsr r r rO Z
get_logger__name__loggerr> boolfloatintrS r0 r0 r0 r3 sj