U +-e@s UddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZmZmZmZmZddlmZm Z ddl!Z!dd l"m#Z#dd l$m%Z%m&Z&dd l'm(Z(d d l)m*Z*d dl(m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6d dl7m8Z8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLmMZMmNZNmOZOmPZPmQZQd dlRmSZSd dlTmUZUd dlVmWZWeOXeYZZe[dZ\iZ]ee^e_fe`d<dlee^edfe_dddZaebZceZde[dZeeddGdddZfeQddddde^e^ee^ee^ee^ee^e^ddd Zgdme^ee^e^d!d"d#Zhdnee^e_ee^e^fd%d&d'Zidddd(ee^ee^eee^dfe^d)d*d+ZjGd,d-d-ekZldoee^d.d/d0Zmdd1d2d3d$d4eWe^eneoeoeeoe_e!jpd5d6d7Zqe!jpd8d9d:Zrdddd3ddd;e^eeoeee^e^feeoeneend<d=d>ZseQddddd$ddd?d$dd$d$d@ e^ee^ee^ee^edfeee^dfe_ee^eeeoe_ee_e^dfe_e_e^dAdBdCZtee^ee^dDdEdFZudpe^e^e_ddGdHdIZvdqe^e^e_ddGdJdKZwe^e^e^ddLdMdNZxeQe^e^e^dOdPdQZyenee^efddRdSdTZzeQdddddddddUdd$ddd?d$dd$d$dVe^e^ee^ee^ee^ee^ee^ee^ee^edfee^edfee_edUfeee^dfe_ee^eeeoe_ee_e^dfe_e_e^dWdXdYZ{eQdre^e^ee^edfee^ee^ee^eddfdZd[d\Z|eQdse^ee_e^dfeeeeoefd]d^d_Z}ee^eend`dadbZ~e^e^ddcdddeZe^e^e^e^dfdgdhZe^e^e^ee_edUfe^didjdkZdS)tN)contextmanager) dataclass)partial)sha256)Path)AnyBinaryIODict GeneratorLiteralOptionalTupleUnion)quoteurlparse)FileLock) ProxyErrorTimeout) constants) __version__) DEFAULT_REVISIONENDPOINTHF_HUB_DISABLE_SYMLINKS_WARNINGHF_HUB_ENABLE_HF_TRANSFERHUGGINGFACE_CO_URL_TEMPLATE HUGGINGFACE_HEADER_X_LINKED_ETAG HUGGINGFACE_HEADER_X_LINKED_SIZE HUGGINGFACE_HEADER_X_REPO_COMMITHUGGINGFACE_HUB_CACHEREPO_ID_SEPARATOR REPO_TYPESREPO_TYPES_URL_PREFIXES)EntryNotFoundErrorGatedRepoErrorLocalEntryNotFoundErrorRepositoryNotFoundErrorRevisionNotFoundErrorSoftTemporaryDirectorybuild_hf_headersget_fastai_versionget_fastcore_versionget_graphviz_versionget_jinja_versionget_pydot_versionget_tf_versionget_torch_versionhf_raise_for_status http_backoffis_fastai_availableis_fastcore_availableis_graphviz_availableis_jinja_availableis_pydot_availableis_tf_availableis_torch_availableloggingtqdmvalidate_hf_hub_args_http_user_agent) _PY_VERSION) HTTP_METHOD_Tzfilename="(?P.*?)";_are_symlinks_supported_in_dir) cache_dirreturnc Cs|dkr t}tt|}|tkrdt|<tj|ddt|d}t|d}| t|d}tj j |tj |d}zt ||WnHtk rdt|<tsd |d }tjd kr|d 7}t|YnXW5QRXt|S) aReturn whether the symlinks are supported on the machine. Since symlinks support can change depending on the mounted disk, we need to check on the precise cache folder. By default, the default HF cache directory is checked. Args: cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. Returns: [bool] Whether symlinks are supported in the directory. NTexist_ok)dirZdummy_file_srcZdummy_file_dst)startFz`huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in a%. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.nta  To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development)rstrr expanduserresolverAosmakedirsr(touchpathrelpathdirnamesymlinkOSErrorrnamewarningswarn)rBZtmpdirZsrc_pathZdst_path relative_srcmessagerY^/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/huggingface_hub/file_download.pyare_symlinks_supportedSs0      r[z^[0-9a-f]{40}$T)frozenc@s>eZdZUdZeeed<eeed<eed<eeed<dS)HfFileMetadataa8Data structure containing information about a file versioned on the Hub. Returned by [`get_hf_file_metadata`] based on a URL. Args: commit_hash (`str`, *optional*): The commit_hash related to the file. etag (`str`, *optional*): Etag of the file on the server. location (`str`): Location where to download the file. Can be a Hub url or not (CDN). size (`size`): Size of the file. In case of an LFS file, contains the size of the actual LFS file, not the pointer. commit_hashetaglocationsizeN)__name__ __module__ __qualname____doc__r rI__annotations__intrYrYrYrZr]s   r] subfolder repo_typerevisionendpoint)repo_idfilenamerirjrkrlrCcCs|dkr d}|dk r"|d|}|tkr2td|tkrFt||}|dkrRt}tj|t|ddt|d}|dk r|tr||t td}|S)a= Construct the URL of a file from the given information. The resolved address can either be a huggingface.co-hosted url, or a link to Cloudfront (a Content Delivery Network, or CDN) for large files which are more than a few MBs. Args: repo_id (`str`): A namespace (user or an organization) name and a repo name separated by a `/`. filename (`str`): The name of the file in the repo. subfolder (`str`, *optional*): An optional value corresponding to a folder inside the repo. repo_type (`str`, *optional*): Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`. revision (`str`, *optional*): An optional Git revision id which can be a branch name, a tag, or a commit hash. endpoint (`str`, *optional*): Hugging Face Hub base url. Will default to https://huggingface.co/. Otherwise, one can set the `HF_ENDPOINT` environment variable. Example: ```python >>> from huggingface_hub import hf_hub_url >>> hf_hub_url( ... repo_id="julien-c/EsperBERTo-small", filename="pytorch_model.bin" ... ) 'https://huggingface.co/julien-c/EsperBERTo-small/resolve/main/pytorch_model.bin' ``` Notes: Cloudfront is replicated over the globe so downloads are way faster for the end user (and it also lowers our bandwidth costs). Cloudfront aggressively caches files by default (default TTL is 24 hours), however this is not an issue here because we implement a git-based versioning system on huggingface.co, which means that we store the files on S3/Cloudfront in a content-addressable way (i.e., the file name is its hash). Using content-addressable filenames means cache can't ever be stale. In terms of client-side caching from this library, we base our caching on the objects' entity tag (`ETag`), which is an identifier of a specific version of a resource [1]_. An object's ETag is: its git-sha1 if stored in git, or its sha256 if stored in git-lfs. References: - [1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag N/zInvalid repo type)safe)rmrkrn) r! ValueErrorr"rrformatr startswithrlen)rmrnrirjrkrlurlrYrYrZ hf_hub_urls$F  rw)rvr_rCcCsN|d}t|}|r8|d}|dt|7}|drJ|d7}|S)aGenerate a local filename from a url. Convert `url` into a hashed filename in a reproducible way. If `etag` is specified, append its hash to the url's, delimited by a period. If the url ends with .h5 (Keras HDF5 weights) adds '.h5' to the name so that TF 2.0 can identify it as a HDF5 file (see https://github.com/tensorflow/tensorflow/blob/00fad90125b18b80fe054de1055770cfb8fe4ba3/tensorflow/python/keras/engine/network.py#L1380) Args: url (`str`): The address to the file. etag (`str`, *optional*): The ETag of the file. Returns: The generated filename. utf-8.z.h5)encoder hexdigestendswith)rvr_Z url_bytesrnZ etag_bytesrYrYrZurl_to_filenames    r}F)rBlegacy_cache_layoutrCc Cs|stdt|dkrt}t|tr.t|}tj ||}tj |sXt d|d|d}tj |s|t d|dt |dd}t |}W5QRX|d}|d }||fS) aW Return the url and etag (which may be `None`) stored for `filename`. Raise `EnvironmentError` if `filename` or its stored metadata do not exist. Args: filename (`str`): The name of the file cache_dir (`str`, *optional*): The cache directory to use instead of the default one. legacy_cache_layout (`bool`, *optional*, defaults to `False`): If `True`, uses the legacy file cache layout i.e. just call `hf_hub_url` then `cached_download`. This is deprecated as the new cache layout is more powerful. z7`filename_to_url` uses the legacy way cache file layoutNzfile z not found.jsonrx)encodingrvr_)rUrV FutureWarningr isinstancerrIrLrOjoinexistsEnvironmentErroropenjsonload) rnrBr~ cache_path meta_path meta_filemetadatarvr_rYrYrZfilename_to_url's(   r library_namelibrary_version user_agent)rrrrCcCst|||dS)z,Deprecated in favor of [`build_hf_headers`].rr=rrYrYrZhttp_user_agentUs rc@s eZdZdS)OfflineModeIsEnabledN)rbrcrdrYrYrYrZrcsrmsgcCs&tjr"t|dkrdn dt|dS)z_Raise a OfflineModeIsEnabled error (subclass of ConnectionError) if HF_HUB_OFFLINE is True.NzOffline mode is enabled.zOffline mode is enabled. )rZHF_HUB_OFFLINErrIrrYrYrZ!_raise_if_offline_mode_is_enabledgsrg?$@) max_retriesbase_wait_time max_wait_timetimeoutfollow_relative_redirects)methodrvrrrrrrCc Kstd||rtf||||||dd|}d|jkrFdkrnnHt|jd} | jdkrtf|t|j| jd||||d d|S|St f|||||t t fd |d |S) aWrapper around requests methods to add several features. What it does: 1. Ensure offline mode is disabled (env variable `HF_HUB_OFFLINE` not set to 1). If enabled, a `OfflineModeIsEnabled` exception is raised. 2. Follow relative redirections if `follow_relative_redirects=True` even when `allow_redirection` kwarg is set to False. 3. Retry in case request fails with a `Timeout` or `ProxyError`, with exponential backoff. Args: method (`str`): HTTP method, such as 'GET' or 'HEAD'. url (`str`): The URL of the resource to fetch. max_retries (`int`, *optional*, defaults to `0`): Maximum number of retries, defaults to 0 (no retries). base_wait_time (`float`, *optional*, defaults to `0.5`): Duration (in seconds) to wait before retrying the first time. Wait time between retries then grows exponentially, capped by `max_wait_time`. max_wait_time (`float`, *optional*, defaults to `2`): Maximum amount of time between two retries, in seconds. timeout (`float`, *optional*, defaults to `10`): How many seconds to wait for the server to send data before giving up which is passed to `requests.request`. follow_relative_redirects (`bool`, *optional*, defaults to `False`) If True, relative redirection (redirection to the same site) will be resolved even when `allow_redirection` kwarg is set to False. Useful when we want to follow a redirection to a renamed repository without following redirection to a CDN. **params (`dict`, *optional*): Params to pass to `requests.request`. zTried to reach F)rrvrrrrr,Locationro)rOTrY)rrvrrrZretry_on_exceptionsZretry_on_status_codesr) r_request_wrapper status_coderheadersnetloc_replacerOgeturlr2rr) rrvrrrrrparamsresponseZ parsed_targetrYrYrZrpsP-    rrCcOs t||S)zDeprecated method. Please use `_request_wrapper` instead. Alias to keep backward compatibility (used in Transformers). )r)argskwargsrYrYrZ_request_with_retrysr)proxies resume_sizerrr expected_size)rv temp_filerrrrrc Cs|strz@ddlm}td|dd} d} |||j| | |dWdStk rdtd Yn,tk r} zt d | W5d} ~ XYnXt |pi}|dkrd |f|d <t d |d||||d} t | | jd} | dk r|t| nd}|}| jd}|dk r0t|}|dk r0|d}t|dkrPd|dd}tdd||d|tttjkd}| jddD]$}|r|t|||q|dk r||krtd|d|d|d|dS)zs Download a remote file. Do not gobble up errors, and will return errors tailored to the Hugging Face Hub. r)downloadz Download z using HF_TRANSFER.di)rNzFast download using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.zAn error occurred while downloading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.z bytes=%d-RangeGETT)rrvstreamrrrrContent-LengthzContent-Dispositionrnu(…)iBz Downloading )unitZ unit_scaletotalinitialdescdisable) chunk_sizez1Consistency check failed: file should be of size z but has size z (z). We are sorry for the inconvenience. Please retry download and pass `force_download=True, resume_download=False` as argument. If the issue persists, please let us know by opening an issue on https://github.com/huggingface/huggingface_hub.) rZ hf_transferrloggerdebugrT ImportErrorrr Exception RuntimeErrorcopydeepcopyrr1rgetrgHEADER_FILENAME_PATTERNsearch groupdictrur;boolgetEffectiveLevelr:NOTSET iter_contentupdatewritetellrclose)rvrrrrrrrrZ max_filesrercontent_lengthrZdisplayed_namecontent_dispositionmatchprogresschunkrYrYrZhttp_getsv        r rrrBrforce_downloadforce_filenamer etag_timeoutresume_downloadtokenlocal_files_onlyr~)rvrrrBrrrrrrrrr~rCc  s| stdt|dkrt}t|tr.t|}tj|ddt | |||d} |}d}d}| sHzd| d<t d|| d d||d }| ddt ||j tp|j d }|dkrtd t|j d }d|jkrdkrnn|j d}| ddd}WnFtjjtjjfk r$Yn$tjjtjjtfk rFYnX|dk rV|nt||}tj||}|dkrtj|r|s|Sddtt ||!dddD}t"|dkr|s|dkrtj||dS| rt#dnt#dtj|r|s|S|d}tj$dkrTt"tj%|dkrTdtj%|}tj$dkrt"tj%|dkrdtj%|}t&|0tj|r|s|W5QRS| r |dt't(t)j*ddfdfd d! }|}tjrt+j,}nd}nt-t.j/d"|d d#}d}|*}t01d$||j$t2||||| |d%W5QRXt01d&||t3|j$||dkrt01d'|||d(}|d)}t4|d*}t56||W5QRXW5QRX|S)+a Download from a given URL and cache it if it's not already present in the local cache. Given a URL, this function looks for the corresponding file in the local cache. If it's not there, download it. Then return the path to the cached file. Will raise errors tailored to the Hugging Face Hub. Args: url (`str`): The path to the file to be downloaded. library_name (`str`, *optional*): The name of the library to which the object corresponds. library_version (`str`, *optional*): The version of the library. cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. user_agent (`dict`, `str`, *optional*): The user-agent info in the form of a dictionary or a string. force_download (`bool`, *optional*, defaults to `False`): Whether the file should be downloaded even if it already exists in the local cache. force_filename (`str`, *optional*): Use this name instead of a generated file name. proxies (`dict`, *optional*): Dictionary mapping protocol to the URL of the proxy passed to `requests.request`. etag_timeout (`float`, *optional* defaults to `10`): When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `requests.request`. resume_download (`bool`, *optional*, defaults to `False`): If `True`, resume a previously interrupted download. token (`bool`, `str`, *optional*): A token to be used for the download. - If `True`, the token is read from the HuggingFace config folder. - If a string, it's used as the authentication token. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, avoid downloading the file and return the path to the local cached file if it exists. legacy_cache_layout (`bool`, *optional*, defaults to `False`): Set this parameter to `True` to mention that you'd like to continue the old cache layout. Putting this to `True` manually will not raise any warning when using `cached_download`. We recommend using `hf_hub_download` to take advantage of the new cache. Returns: Local path (string) of file or if networking is off, last version of file cached on disk. Raises the following errors: - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `token=True` and the token cannot be found. - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined. - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid - [`~utils.RepositoryNotFoundError`] If the repository to download from cannot be found. This may be because it doesn't exist, or because it is set to `private` and you do not have access. - [`~utils.RevisionNotFoundError`] If the revision to download from cannot be found. - [`~utils.EntryNotFoundError`] If the file to download cannot be found. - [`~utils.LocalEntryNotFoundError`] If network is disabled or unavailable and file is not found in cache. zu'cached_download' is the legacy way to download files from the HF hub, please consider upgrading to 'hf_hub_download'NTrDrrrridentityAccept-EncodingHEADFrrvrallow_redirectsrrrETag\Distant resource does not have an ETag, we won't be able to reliably ensure reproducibility.rrrr authorizationcSs$g|]}|ds|ds|qS)r.lock)r|).0filerYrYrZ s z#cached_download..ryrz.*zCannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.zConnection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.rrH\\?\ .incompleterc 3s td }|VW5QRXdSNabrfZincomplete_pathrYrZ_resumable_file_managers z0cached_download.._resumable_file_managerwbmoderFdeletedownloading %s to %srrrrzstoring %s in cache at %szcreating metadata file for %s)rvr_rw)7rUrVrrrrrIrLrMr)rpopr1rrrrS _int_or_nonerrequests exceptionsSSLErrorrConnectionErrorrrr}rOrrfnmatchfilterlistdirsplitrur%rTabspathrrr ioBufferedWriterstatst_sizertempfileNamedTemporaryFilerinfor_chmod_and_replacerrdump)rvrrrBrrrrrrrrr~rurl_to_downloadr_rrrnrZmatching_files lock_pathrtemp_file_managerrrmetarrrYrrZcached_download7s[     ""         r)r_rCcCs|dkr dS|ddS)aONormalize ETag HTTP header, so it can be used to create nice filepaths. The HTTP spec allows two forms of ETag: ETag: W/"" ETag: "" For now, we only expect the second form from the server, but we want to be future-proof so we support both. For more context, see `TestNormalizeEtag` tests and https://github.com/huggingface/huggingface_hub/pull/1428. Args: etag (`str`, *optional*): HTTP header Returns: `str` or `None`: string that can be used as a nice directory name. Returns `None` if input is None. NzW/")lstripstrip)r_rYrYrZ_normalize_etag>sr)srcdstnew_blobrCcCst|||dS)z6Alias method used in `transformers` conversion script.rr r!)_create_symlinkr"rYrYrZ_create_relative_symlinkTsr$c Cszt|Wntk r"YnXtjtj|}tjtj|}ztj|tj|}Wntk r~d}YnXzHz$tj ||g}t tj|}Wntk rtj dk}YnXWn$t k rt tj|}YnX|rh|p|}t d|d|zt||WnBtk rdtj|r^tj|tj|kr^nYnXnL|rt d|d|t||n"t d|d|t||dS)uCreate a symbolic link named dst pointing to src. By default, it will try to create a symlink using a relative path. Relative paths have 2 advantages: - If the cache_folder is moved (example: back-up on a shared drive), relative paths within the cache folder will not brake. - Relative paths seems to be better handled on Windows. Issue was reported 3 times in less than a week when changing from relative to absolute paths. See https://github.com/huggingface/huggingface_hub/issues/1398, https://github.com/huggingface/diffusers/issues/2729 and https://github.com/huggingface/transformers/pull/22228. NOTE: The issue with absolute paths doesn't happen on admin mode. When creating a symlink from the cache to a local folder, it is possible that a relative path cannot be created. This happens when paths are not on the same volume. In that case, we use absolute paths. The result layout looks something like └── [ 128] snapshots ├── [ 128] 2439f60ef33a0d46d85da5001d52aeda5b00ce9f │ ├── [ 52] README.md -> ../../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812 │ └── [ 76] pytorch_model.bin -> ../../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd If symlinks cannot be created on this platform (most likely to be Windows), the workaround is to avoid symlinks by having the actual file in `dst`. If it is a new file (`new_blob=True`), we move it to `dst`. If it is not a new file (`new_blob=False`), we don't know if the blob file is already referenced elsewhere. To avoid breaking existing cache, the file is duplicated on the disk. In case symlinks are not supported, a warning message is displayed to the user once when loading `huggingface_hub`. The warning message can be disable with the `DISABLE_SYMLINKS_WARNING` environment variable. NrHzCreating pointer from z to z(Symlink not supported. Moving file from z)Symlink not supported. Copying file from )rLremoverSrOr rJrPrQrr commonpathr[rTPermissionErrorrrrRFileExistsErrorislinkrealpathshutilmovecopyfile) rr r!Zabs_srcZabs_dstrWr&Z_support_symlinksZsrc_rel_or_absrYrYrZr#Ys@ ( r#)storage_folderrkr^rCcCsJ||krFt|d|}|jjddd|r<||krF||dS)zCache reference between a revision (tag, branch or truncated commit hash) and the corresponding commit hash. Does nothing if `revision` is already a proper `commit_hash` or reference is already cached. refsTparentsrEN)rparentmkdirr read_text write_text)r.rkr^ref_pathrYrYrZ(_cache_commit_hash_for_specific_revisions r7)rmrjrCcCs |df|d}t|S)zReturn a serialized version of a hf.co repo name and type, safe for disk storage as a single non-nested folder. Example: models--julien-c--EsperBERTo-small srp)r r r)rmrjpartsrYrYrZrepo_folder_namesr:)r target_dirrCc Cs~t|}|gt|jD]`}zFt|j}||kr\td|ddd|d|dddWdStk rvYqXqdS)a$Check disk usage and log a warning if there is not enough disk space to download the file. Args: expected_size (`int`): The expected size of the file in bytes. target_dir (`str`): The directory where the file will be stored after downloading. zLNot enough free disk space to download the file. The expected file size is: g.Az.2fz MB. The target location z only has z MB free disk space.N) rlistr1r+ disk_usagefreerUrVrS)rr;rOZtarget_dir_freerYrYrZ_check_disk_spaces  "r?auto)rirjrkrlrrrB local_dirlocal_dir_use_symlinksrrrrrrrrr~)rmrnrirjrkrlrrrBrArBrrrrrrrrr~rCc,s | dk rtdtd}|rRt||||||d}t||||| | | ||||||d S|dkr^t}|dkrjt}t|tr|t |}t| trt | } |dkrd}|dk r|d|}|dkrd}|t krt d |d t t t j |t||d }t j|dd t j j|d}t jd krD|ds4d|krDt d|dt|rt|||}t j |r| dk rt|| || dS|St|||||d}t|||| d}|}d}d}d}d}|sJzzt||||d}Wn|tk rV}z\|jjt}|dk rD|sDt|d||} | j j!ddd| "t#|||W5d}~XYnX|j$}|dkrpt%d|j&}|dkrt%d|j'}|j(|kr|j(}|)ddWnt*j+j,t*j+j-fk rYnxt*j+j.t*j+j/t0fk r}!z|!}W5d}!~!XYnDt1tfk r Yn*t*j2k rH}!z|!}W5d}!~!XYnX|dkr,| rbt dd}t|rx|}n:t j |d|}"t j 3|"rt4|"}#|#5}W5QRX|dk rt|||}t j |r| dk rt|| || dS|S|rt6dn(t|t7st|t8r"|n t6d||dk s>t9d|dk sPt9d t j |d!|}$t|||}t jt j :|$dd t jt j :|dd t#|||t j |r| s| dk rt|| || dS|St j |$r| s| dk rt|$| || dSt;|$|d"d#|S|$d$}%t jd krPt|%Nt j |r| s|W5QRS|r|$d't?t@tAjBddfd(fd)d* }&|&}'t j rt CjD}(nd+}(ntEtFjGd,|d"d-}'d+}(|'n})tHId.||)j|dk rztJ|t j :|)jtJ|t j :|$| dk rztJ|| tK||)||(||d/W5QRX| dkrtHId0|d1|$tL|)j|$t;|$|dd#nt j | |}*t jt j :|*dd t C|)jjDtMjNk}+| dks(| d2krd|+rdtHId0|d1|$tL|)j|$tHId3t;|$|*d"d#nn| d2kr|+stHId0|d1|$tL|)j|$tHId4tOP|$|*n$tHId0|d5|*d6tL|)j|*|*}W5QRXzt Q|%Wnt%k rYnX|S)7uDownload a given file if it's not already present in the local cache. The new cache file layout looks like this: - The cache directory contains one subfolder per repo_id (namespaced by repo type) - inside each repo folder: - refs is a list of the latest known revision => commit_hash pairs - blobs contains the actual file blobs (identified by their git-sha or sha256, depending on whether they're LFS files or not) - snapshots contains one subfolder per commit, each "commit" contains the subset of the files that have been resolved at that particular commit. Each filename is a symlink to the blob at that particular commit. If `local_dir` is provided, the file structure from the repo will be replicated in this location. You can configure how you want to move those files: - If `local_dir_use_symlinks="auto"` (default), files are downloaded and stored in the cache directory as blob files. Small files (<5MB) are duplicated in `local_dir` while a symlink is created for bigger files. The goal is to be able to manually edit and save small files without corrupting the cache while saving disk space for binary files. The 5MB threshold can be configured with the `HF_HUB_LOCAL_DIR_AUTO_SYMLINK_THRESHOLD` environment variable. - If `local_dir_use_symlinks=True`, files are downloaded, stored in the cache directory and symlinked in `local_dir`. This is optimal in term of disk usage but files must not be manually edited. - If `local_dir_use_symlinks=False` and the blob files exist in the cache directory, they are duplicated in the local dir. This means disk usage is not optimized. - Finally, if `local_dir_use_symlinks=False` and the blob files do not exist in the cache directory, then the files are downloaded and directly placed under `local_dir`. This means if you need to download them again later, they will be re-downloaded entirely. ``` [ 96] . └── [ 160] models--julien-c--EsperBERTo-small ├── [ 160] blobs │ ├── [321M] 403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd │ ├── [ 398] 7cb18dc9bafbfcf74629a4b760af1b160957a83e │ └── [1.4K] d7edf6bd2a681fb0175f7735299831ee1b22b812 ├── [ 96] refs │ └── [ 40] main └── [ 128] snapshots ├── [ 128] 2439f60ef33a0d46d85da5001d52aeda5b00ce9f │ ├── [ 52] README.md -> ../../blobs/d7edf6bd2a681fb0175f7735299831ee1b22b812 │ └── [ 76] pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd └── [ 128] bbc77c8132af1cc5cf678da3f1ddf2de43606d48 ├── [ 52] README.md -> ../../blobs/7cb18dc9bafbfcf74629a4b760af1b160957a83e └── [ 76] pytorch_model.bin -> ../../blobs/403450e234d65943a7dcf7e05a771ce3c92faa84dd07db4ac20f592037a1e4bd ``` Args: repo_id (`str`): A user or an organization name and a repo name separated by a `/`. filename (`str`): The name of the file in the repo. subfolder (`str`, *optional*): An optional value corresponding to a folder inside the model repo. repo_type (`str`, *optional*): Set to `"dataset"` or `"space"` if downloading from a dataset or space, `None` or `"model"` if downloading from a model. Default is `None`. revision (`str`, *optional*): An optional Git revision id which can be a branch name, a tag, or a commit hash. endpoint (`str`, *optional*): Hugging Face Hub base url. Will default to https://huggingface.co/. Otherwise, one can set the `HF_ENDPOINT` environment variable. library_name (`str`, *optional*): The name of the library to which the object corresponds. library_version (`str`, *optional*): The version of the library. cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. local_dir (`str` or `Path`, *optional*): If provided, the downloaded file will be placed under this directory, either as a symlink (default) or a regular file (see description for more details). local_dir_use_symlinks (`"auto"` or `bool`, defaults to `"auto"`): To be used with `local_dir`. If set to "auto", the cache directory will be used and the file will be either duplicated or symlinked to the local directory depending on its size. It set to `True`, a symlink will be created, no matter the file size. If set to `False`, the file will either be duplicated from cache (if already exists) or downloaded from the Hub and not cached. See description for more details. user_agent (`dict`, `str`, *optional*): The user-agent info in the form of a dictionary or a string. force_download (`bool`, *optional*, defaults to `False`): Whether the file should be downloaded even if it already exists in the local cache. proxies (`dict`, *optional*): Dictionary mapping protocol to the URL of the proxy passed to `requests.request`. etag_timeout (`float`, *optional*, defaults to `10`): When fetching ETag, how many seconds to wait for the server to send data before giving up which is passed to `requests.request`. resume_download (`bool`, *optional*, defaults to `False`): If `True`, resume a previously interrupted download. token (`str`, `bool`, *optional*): A token to be used for the download. - If `True`, the token is read from the HuggingFace config folder. - If a string, it's used as the authentication token. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, avoid downloading the file and return the path to the local cached file if it exists. legacy_cache_layout (`bool`, *optional*, defaults to `False`): If `True`, uses the legacy file cache layout i.e. just call [`hf_hub_url`] then `cached_download`. This is deprecated as the new cache layout is more powerful. Returns: Local path (string) of file or if networking is off, last version of file cached on disk. Raises the following errors: - [`EnvironmentError`](https://docs.python.org/3/library/exceptions.html#EnvironmentError) if `token=True` and the token cannot be found. - [`OSError`](https://docs.python.org/3/library/exceptions.html#OSError) if ETag cannot be determined. - [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) if some parameter value is invalid - [`~utils.RepositoryNotFoundError`] If the repository to download from cannot be found. This may be because it doesn't exist, or because it is set to `private` and you do not have access. - [`~utils.RevisionNotFoundError`] If the revision to download from cannot be found. - [`~utils.EntryNotFoundError`] If the file to download cannot be found. - [`~utils.LocalEntryNotFoundError`] If network is disabled or unavailable and file is not found in cache. NzThe `force_filename` parameter is deprecated as a new caching system, which keeps the filenames as they are on the Hub, is now in place.TrhrrorpmodelInvalid repo type: . Accepted repo types are: )rmrjrDrHz..\z\..\z*Invalid filename: cannot handle filename 'zB' on Windows. Please ask the repository owner to rename this file.) use_symlinks)rjrkrlr)rvrrr .no_existr0zODistant resource does not seem to be on huggingface.co (missing commit header).rrzbWe have no connection or you passed local_files_only, so force_download is not an accepted option.r/zCannot find the requested files in the disk cache and outgoing traffic has been disabled. To enable hf.co look-ups and downloads online, set 'local_files_only' to False.zAn error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.z)etag must have been retrieved from serverz0commit_hash must have been retrieved from serverZblobsFr!rrrrrc 3s td }|VW5QRXdSrrrrrYrZrys z0hf_hub_download.._resumable_file_managerrrrrrzStoring z in cache at r@zCreate symlink to local dirzADuplicate in local dir (small file and use_symlink set to 'auto')z in local_dir at z (not cached).)RrUrVrrwrrrrrrIr!rrrLrOrr:rMr rTrtREGEX_COMMIT_HASHr_get_pointer_pathr _to_local_dirr)get_hf_file_metadatar#rrrrr2r3rNr7r^rSr_rar`rrrrrrrrr' HTTPErrorisfilerreadr%r&r$AssertionErrorrQr#rur rrr r rrrrrrrrr?rrr'HF_HUB_LOCAL_DIR_AUTO_SYMLINK_THRESHOLDr+r-r%),rmrnrirjrkrlrrrBrArBrrrrrrrrr~rvr.relative_filename pointer_pathrrr_r^rZhead_call_errorr http_errorZno_exist_file_patherrorr6rZ blob_pathrrrrrlocal_dir_filepathZ is_big_filerYrrZhf_hub_downloads                       ""            rW)rmrnrBrkrjrCc CsR|dkr d}|dkrd}|tkr8td|dtt|dkrDt}|dd}tj||d|}tj|sxdStj|d }tj|d }tj|d } tj|rtj||} tj | rt | } | }W5QRXtj tj| ||rt Stj |sdSt|} || kr,dStj|||} tj | rN| SdS) a Explores the cache to return the latest cached file for a given revision if found. This function will not raise any exception if the file in not cached. Args: cache_dir (`str` or `os.PathLike`): The folder where the cached files lie. repo_id (`str`): The ID of the repo on huggingface.co. filename (`str`): The filename to look for inside `repo_id`. revision (`str`, *optional*): The specific model version to use. Will default to `"main"` if it's not provided and no `commit_hash` is provided either. repo_type (`str`, *optional*): The type of the repository. Will default to `"model"`. Returns: `Optional[str]` or `_CACHED_NO_EXIST`: Will return `None` if the file was not cached. Otherwise: - The exact path to the cached file if it's found in the cache - A special value `_CACHED_NO_EXIST` if the file does not exist at the given commit hash and this fact was cached. Example: ```python from huggingface_hub import try_to_load_from_cache, _CACHED_NO_EXIST filepath = try_to_load_from_cache() if isinstance(filepath, str): # file exists and is cached ... elif filepath is _CACHED_NO_EXIST: # non-existence of file is cached ... else: # file is not cached ... ``` NmainrCrDrErpz--zs--r/ snapshotsrG)r!rrrIrreplacerLrOrisdirrNrrO_CACHED_NO_EXISTrr )rmrnrBrkrjZ object_idZ repo_cacheZrefs_dirZ snapshots_dirZ no_exist_dirZ revision_filerZ cached_shasZ cached_filerYrYrZtry_to_load_from_caches:2       r])rvrrrrCc Cst|d}d|d<td||dd||d}t|t|jtt|jtpT|jd|jd ph|j j t |jt p|jd d S) aFetch metadata of a file versioned on the Hub for a given url. Args: url (`str`): File url, for example returned by [`hf_hub_url`]. token (`str` or `bool`, *optional*): A token to be used for the download. - If `True`, the token is read from the HuggingFace config folder. - If `False` or `None`, no token is provided. - If a string, it's used as the authentication token. proxies (`dict`, *optional*): Dictionary mapping protocol to the URL of the proxy passed to `requests.request`. timeout (`float`, *optional*, defaults to 10): How many seconds to wait for the server to send metadata before giving up. Returns: A [`HfFileMetadata`] object containing metadata such as location, etag, size and commit_hash. )rrrrFTrrrr)r^r_r`ra) r)rr1r]rrrrrrequestrvrr)rvrrrrrrYrYrZrLs*      rL)valuerCc Cs*z t|WSttfk r$YdSXdS)N)rg TypeErrorrr)r_rYrYrZr[s r)rr rCcCsbt|jjdt}z,|t|j}t |t |W5|Xt ||dS)aVSet correct permission before moving a blob from tmp directory to cache dir. Do not take into account the `umask` from the process as there is no convenient way to get it that is thread-safe. See: - About umask: https://docs.python.org/3/library/os.html#os.umask - Thread-safety: https://stackoverflow.com/a/70343066 - About solution: https://github.com/huggingface/huggingface_hub/pull/1220#issuecomment-1326211591 - Fix issue: https://github.com/huggingface/huggingface_hub/issues/1141 - Fix issue: https://github.com/huggingface/huggingface_hub/issues/1215 Ztmp_N) rr2uuiduuid4unlinkrNrst_moderLchmodS_IMODEr+r,)rr Ztmp_fileZcache_dir_moderYrYrZrbs r)r.rkrRrCcCs`tj|d}tj|||}ttj|ttj|jkr\td|d|d|d|S)NrYzXInvalid pointer path: cannot create pointer path in snapshot folder if `storage_folder='z'`, `revision='z'` and `relative_filename='z'`.)rLrOrrr r1rr)r.rkrRZ snapshot_pathrSrYrYrZrJ{s"rJ)rOrArRrFrCcCstj||}ttj|ttj|jkrFtd|d|dtjtj|ddtj |}|dkrt |j t j k}|rt||ddn t|||S) zPlace a file in a local dir (different than cache_dir). Either symlink to blob file in cache or duplicate file depending on `use_symlinks` and file size. zCannot copy file 'z' to local dir 'z,': file would not be in the local directory.TrDr@FrH)rLrOrrr r1rrrMrQr*rrrrQr#r+r-)rOrArRrFrVZreal_blob_pathrYrYrZrKs"  rK)N)N)NF)N)F)F)NNN)NNr)rrr rrLrer+rrrarU contextlibr dataclassesr functoolsrhashlibrpathlibrtypingrrr r r r r r urllib.parserrrZfilelockrZrequests.exceptionsrrZhuggingface_hubrrorrrrrrrrrrr r!r"utilsr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<Zutils._headersr>Zutils._runtimer?Z utils._typingr@Z get_loggerrbrcompilerrArIrrfr[objectr\Z_CACHED_NO_EXIST_TrIr]rwr}rrrrrrgfloatResponserrrrrr$r#r7r:r?rWr]rLrrrJrKrYrYrYrZs     (   8p     < [! 0    a  ^   P     g  \ ;