U +-ed @sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZdd lmZdd lm Z d d l!m"Z"m#Z#d d l$m%Z%m&Z&m'Z'd dl(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/d dl(m0Z1erd dl2m3Z3e-4e5Z6edZ7dZ8e GdddZ9e GdddZ:e GdddZ;eee>ddddZ?e/ddd ee;eeer)r2r4r4r5r6is z!CommitOperationCopy.__post_init__) r7r8r9r:r;r<r?rr6r4r4r4r5r=Ps r=c@sxeZdZUdZeed<eeeee fed<e dddZ e ed<ddd d Z edeee d d d ZedddZdS)CommitOperationAdda Data structure holding necessary info to upload a file to a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` path_or_fileobj (`str`, `Path`, `bytes`, or `BinaryIO`): Either: - a path to a local file (as `str` or `pathlib.Path`) to upload - a buffer of bytes (`bytes`) holding the content of the file to upload - a "file object" (subclass of `io.BufferedIOBase`), typically obtained with `open(path, "rb")`. It must support `seek()` and `tell()` methods. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is not one of `str`, `Path`, `bytes` or `io.BufferedIOBase`. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `str` or `Path` but not a path to an existing file. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `io.BufferedIOBase` but it doesn't support both `seek()` and `tell()`. r)path_or_fileobjF)initrepr upload_infoNreturnc Cs(t|j|_t|jtr$t|j|_t|jtrdtjtj |j}tj |s~t d|dnt|jt j tfs~t dt|jt j rz|j|jdtjWn0ttfk r}zt d|W5d}~XYnXt|jtrt|j|_n,t|jtrt|j|_nt|j|_dS)z6Validates `path_or_fileobj` and compute `upload_info`.zProvided path: 'z(' is not a file on the local file systemzpath_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.rzNpath_or_fileobj is a file-like object but does not implement seek() and tell()N)r-r)r/rBrr;ospathnormpath expanduserisfiler1ioBufferedIOBasebytestellseekSEEK_CUROSErrorAttributeErrorr from_pathrE from_bytesZ from_fileobj)r3rBexcr4r4r5r6s4       z CommitOperationAdd.__post_init__) with_tqdmrGc cst|jtst|jtrZ|r:t|j }|VW5QRXqt|jd }|VW5QRXnLt|jtrvt|jVn0t|jtj r|j }|jV|j |tj dS)u A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. Args: with_tqdm (`bool`, *optional*, defaults to `False`): If True, iterating over the file object will display a progress bar. Only works if the file-like object is a path to a file. Pure bytes and buffers are not supported. Example: ```python >>> operation = CommitOperationAdd( ... path_in_repo="remote/dir/weights.h5", ... path_or_fileobj="./local/weights.h5", ... ) CommitOperationAdd(path_in_repo='remote/dir/weights.h5', path_or_fileobj='./local/weights.h5') >>> with operation.as_file() as file: ... content = file.read() >>> with operation.as_file(with_tqdm=True) as file: ... while True: ... data = file.read(1024) ... if not data: ... break config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] >>> with operation.as_file(with_tqdm=True) as file: ... requests.put(..., data=file) config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] ``` rbN) r/rBr;rr!openrOrMBytesIOrNrPrQSEEK_SET)r3rXfileZprev_posr4r4r5as_files$   zCommitOperationAdd.as_filec Cs.|}t|W5QRSQRXdS)z[ The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)r^base64 b64encoderead)r3r]r4r4r5 b64contents zCommitOperationAdd.b64content)F)r7r8r9r:r;r<rrrOr rrErr6rr0rr^rbr4r4r4r5rAns #1rA)r)rGcCs|dr|dd}|dks0|dks0|dr@td|d|drV|d d}td d |dDr~td |d |S)Nr,r.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./css|]}|dkVqdS)z.gitNr4).0partr4r4r5 sz)_validate_path_in_repo..z^Invalid `path_in_repo` in CommitOperation: cannot update files under a '.git/' folder (path: 'z').) startswithr1anysplit)r)r4r4r5r-s     r-) operationsrGcCstt}|D]}|j}t|trr||dkr>td|d||d7<t|jD]}|t |d7<qXt|t r |t t|dkr |j rtd|dq td|dq dS)a Warn user when a list of operations is expected to overwrite itself in a single commit. Rules: - If a filepath is updated by multiple `CommitOperationAdd` operations, a warning message is triggered. - If a filepath is updated at least once by a `CommitOperationAdd` and then deleted by a `CommitOperationDelete`, a warning is triggered. - If a `CommitOperationDelete` deletes a filepath that is then updated by a `CommitOperationAdd`, no warning is triggered. This is usually useless (no need to delete before upload) but can happen if a user deletes an entire folder and then add new files to it. rzBAbout to update multiple times the same file in the same commit: 'z9'. This can cause undesired inconsistencies in your repo.rz_About to delete a folder containing files that have just been updated within the same commit: 'zLAbout to delete a file that have just been updated within the same commit: 'N) rintr)r/rAwarningswarnrparentsr;r(r+)rlZnb_additions_per_path operationr)parentr4r4r5warn_on_overwriting_operationss(      rs)endpoint num_threads) additions repo_typerepo_idtokenrurvcs~g}t|ddD]R}tdd|D|||d\}} | rZddd| D} td| ||7}qd d |Dg} |D]:} | d d krtd | djdqz| | qzt | dkrtdd Sd dfdd } t rtdt | dt | D]} | | qn`t | dkr@td| | dn:tdt | d|dt | | dt | d|t dd S)a Uploads the content of `additions` to the Hub using the large file storage protocol. Relevant external documentation: - LFS Batch API: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) num_threads (`int`, *optional*): The number of concurrent threads to use when uploading. Defaults to 5. Raises: `RuntimeError` if an upload failed for any reason Raises: `ValueError` if the server returns malformed responses Raises: `requests.HTTPError` if the LFS batch endpoint returned an HTTP error ) chunk_sizecSsg|] }|jqSr4)rErfopr4r4r5 Wsz$upload_lfs_files..)Z upload_infosrzryrxru c Ss0g|](}d|dd|didqS)z$Encountered error for file with OID oidz: `errormessage)get)rferrr4r4r5rasz$LFS batch endpoint returned errors: cSsi|]}|jj|qSr4)rEsha256hex)rfZadd_opr4r4r5 is z$upload_lfs_files..actionsNzContent of file rz/ is already present upstream - skipping upload.rzNo LFS files to upload.rFc sZz|d}t||dWn6tk rT}ztd|jd|W5d}~XYnXdS)Nr)rqZlfs_batch_actionrzzError while uploading 'z ' to the Hub.)r Exception RuntimeErrorr))Z batch_actionrqrWZ oid2addoprzr4r5_wrapped_lfs_upload{s  z-upload_lfs_files.._wrapped_lfs_uploadz Uploading z* LFS files to the Hub using `hf_transfer`.rzUploading 1 LFS file to the Hubz" LFS files to the Hub using up to z threads concurrentlyzUpload z LFS files)desc max_workersZ tqdm_class) rrjoinr1rloggerdebugr)appendlenrhf_tqdmr)rwrxryrzrurvZ batch_actionschunkZbatch_actions_chunkZbatch_errors_chunkrZfiltered_actionsactionrr4rr5upload_lfs_files+s\)        r)preupload_infocCsh|d}t|tstd|D]B}t|trZt|dtrZt|dtrZ|ddks tdq |S)Nfilesz&preupload_info is improperly formattedrI uploadModer%z'preupload_info is improperly formatted:)rr/listr1dictr;)rr file_infor4r4r5_validate_preupload_infos    rF)rwrxryrzrevisionru create_prrGc Cs|dk r |nt}t|d}i}t|dD]z} ddd| Di} tj|d|d|d || ||rld d indd } t| t| } |jfd d| dDq(|D]} | j j dkr| j }d||<q|S)av Requests the Hub "preupload" endpoint to determine whether each input file should be uploaded as a regular git blob or as git LFS blob. Args: additions (`Iterable` of :class:`CommitOperationAdd`): Iterable of :class:`CommitOperationAdd` describing the files to upload to the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[str, UploadMode]` Key is the file path, value is the upload mode ("regular" or "lfs"). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. N)rzr{rcSs8g|]0}|jt|jjd|jj|jjdqS)ascii)rIsamplesizesha) r)r_r`rErdecoderrrr}r4r4r5rs  z&fetch_upload_modes..z/api/zs/z /preupload/r1)jsonheadersparamscSsi|]}|d|dqS)rIrr4)rfr]r4r4r5rsz&fetch_upload_modes..rr') rrrrpostrrrupdaterErr))rwrxryrzrrurr upload_modesrpayloadresprZadditionrIr4r4r5fetch_upload_modess.%     rr$)copiesrxryrzrrurGcCsddlm}|||d}i}t|dddD]\} } t| } dd| D} td t| tD]L} |j|| | | t| px||d } | D] }|jst d |||j | f<qqZ| D].}|j | f|krt d |j d | p|dqq,|S)a` Requests the Hub files information of the LFS files to be copied, including their sha256. Args: copies (`Iterable` of :class:`CommitOperationCopy`): Iterable of :class:`CommitOperationCopy` describing the files to copy on the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[Tuple[str, Optional[str]], RepoFile]]` Key is the file path and revision of the file to copy, value is the repo file. Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. r)HfApi)rurzcSs|jSr@)r?)r~r4r4r5z)fetch_lfs_files_to_copy..)keycSsg|] }|jqSr4)r>r}r4r4r5rsz+fetch_lfs_files_to_copy..r)rypathsrrx)Copying a non-LFS file is not implementedz Cannot copy z at revision z: file is missing on repo.) hf_apirrrrangerFETCH_LFS_BATCH_SIZEZlist_files_infor&NotImplementedErrorZ rfilenamer>r)rrxryrzrrurr files_to_copyr?rlroffsetZsrc_repo_filesZ src_repo_filerqr4r4r5fetch_lfs_files_to_copys.#  r)rlrrcommit_messagecommit_description parent_commitrGc csN|dk r |nd}||d}|dk r*||d<d|dV|D] }t|trz||jdkrzd||jd d dVq:t|tr||jd krd |jd |jj|jj ddVq:t|t r|j rdndd|jidVq:t|t r,||j |jf}|jstdd |jd |jd ddVq:td|d||jq:dS)aG Builds the payload to POST to the `/commit` API of the Hub. Payload is returned as an iterator so that it can be streamed as a ndjson in the POST request. For more information, see: - https://github.com/huggingface/huggingface_hub/issues/1085#issuecomment-1265208073 - http://ndjson.org/ N)summary descriptionZ parentCommitheader)rvaluer'r]r_)contentrIencodingr&ZlfsFiler)rIalgorrZ deletedFolderZ deletedFilerIr)rIrrz(Unknown operation to commit. Operation: z. Upload mode: )r/rArr)rbrrErrrr(r+r=r>r?r&rr1) rlrrrrr header_valuerqZ file_to_copyr4r4r5prepare_commit_payload.sL           r)NF)N)NN)Hr:r_rMrHrn collectionsr contextlibr dataclassesrr itertoolsrpathlibrrtypingr r r r r rrrrrrZtqdm.contrib.concurrentrZhuggingface_hubr constantsrrr&rrrutilsrrrrr r!r"r#rrr$Z get_loggerr7rZ UploadModerr(r=rAr;r-ZCommitOperationrsrmrrrr0rrrr4r4r4r5s   4  $   {.j JB