U +-e0 @s~dZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z m Z mZddlmZmZddlmZdd lmZdd lmZerdd lmZGd d d eZdZdZdZdZdZ ej!dej"dZ#ed(e eeefe$e$e e e ee e efdddZ%eGdddZ&eGdddZ'de(e(e e(e'e e(e e(edd d!Z)e(e e(e'e(d"d#d$Z*e(e e(d%d&d'Z+dS))zLContains utilities to multi-commits (i.e. push changes iteratively on a PR).N) dataclassfield)sha256) TYPE_CHECKINGIterableListOptionalSetTupleUnion)CommitOperationAddCommitOperationDelete)DiscussionWithDetails) experimental) _format_size)HfApic@seZdZdZdS)MultiCommitExceptionzFBase exception for any exception happening while doing a multi-commit.N)__name__ __module__ __qualname____doc__rr_/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/huggingface_hub/_multi_commits.pyrsra ## {commit_message} {commit_description} **Multi commit ID:** {multi_commit_id} Scheduled commits: {multi_commit_strategy} _This is a PR opened using the `huggingface_hub` library in the context of a multi-commit. PR can be commented as a usual PR. However, please be aware that manually updating the PR description, changing the PR status, or pushing new commits, is not recommended as it might corrupt the commit process. Learn more about multi-commits [in this guide](https://huggingface.co/docs/huggingface_hub/main/guides/upload)._ am Multi-commit is now completed! You can ping the repo owner to review the changes. This PR can now be commented or modified without risking to corrupt it. _This is a comment posted using the `huggingface_hub` library in the context of a multi-commit. Learn more about multi-commits [in this guide](https://huggingface.co/docs/huggingface_hub/main/guides/upload)._ a `create_pr=False` has been passed so PR is automatically merged. _This is a comment posted using the `huggingface_hub` library in the context of a multi-commit. Learn more about multi-commits [in this guide](https://huggingface.co/docs/huggingface_hub/main/guides/upload)._ a2 Cannot merge Pull Requests as no changes are associated. This PR will be closed automatically. _This is a comment posted using the `huggingface_hub` library in the context of a multi-commit. Learn more about multi-commits [in this guide](https://huggingface.co/docs/huggingface_hub/main/guides/upload)._ a  An error occurred while trying to merge the Pull Request: `{error_message}`. _This is a comment posted using the `huggingface_hub` library in the context of a multi-commit. Learn more about multi-commits [in this guide](https://huggingface.co/docs/huggingface_hub/main/guides/upload)._ z8- \[(?P[ |x])\].*(?P[a-fA-F0-9]{64}))flags2) operationsmax_operations_per_commitmax_upload_size_per_commitreturnc Csg}g}g}d}g}|D]}t|trL||t||kr||g}nZ|jj|krf||gn@||jj|kr||||jj7}n|||g}|jj}t||kr||g}d}qt|dkr||t|dkr||||fS)a( Split a list of operations in a list of commits to perform. Implementation follows a sub-optimal (yet simple) algorithm: 1. Delete operations are grouped together by commits of maximum `max_operations_per_commits` operations. 2. All additions exceeding `max_upload_size_per_commit` are committed 1 by 1. 3. All remaining additions are grouped together and split each time the `max_operations_per_commit` or the `max_upload_size_per_commit` limit is reached. We do not try to optimize the splitting to get the lowest number of commits as this is a NP-hard problem (see [bin packing problem](https://en.wikipedia.org/wiki/Bin_packing_problem)). For our use case, it is not problematic to use a sub-optimal solution so we favored an easy-to-explain implementation. Args: operations (`List` of [`~hf_api.CommitOperation`]): The list of operations to split into commits. max_operations_per_commit (`int`): Maximum number of operations in a single commit. Defaults to 50. max_upload_size_per_commit (`int`): Maximum size to upload (in bytes) in a single commit. Defaults to 2GB. Files bigger than this limit are uploaded, 1 per commit. Returns: `Tuple[List[List[CommitOperationAdd]], List[List[CommitOperationDelete]]]`: a tuple. First item is a list of lists of [`CommitOperationAdd`] representing the addition commits to push. The second item is a list of lists of [`CommitOperationDelete`] representing the deletion commits. `plan_multi_commits` is experimental. Its API and behavior is subject to change in the future without prior notice. Example: ```python >>> from huggingface_hub import HfApi, plan_multi_commits >>> addition_commits, deletion_commits = plan_multi_commits( ... operations=[ ... CommitOperationAdd(...), ... CommitOperationAdd(...), ... CommitOperationDelete(...), ... CommitOperationDelete(...), ... CommitOperationAdd(...), ... ], ... ) >>> HfApi().create_commits_on_pr( ... repo_id="my-cool-model", ... addition_commits=addition_commits, ... deletion_commits=deletion_commits, ... (...) ... verbose=True, ... ) ``` The initial order of the operations is not guaranteed! All deletions will be performed before additions. If you are not updating multiple times the same file, you are fine. r) isinstancerappendlen upload_infosize) rrraddition_commitsdeletion_commits additionsZadditions_sizeZ deletionsoprrrplan_multi_commitsMs8B             r*c@s`eZdZUdZeeeefed<e ddZ e ed<dZ e ed<ddd d Ze dd d ZdS) MultiCommitStepaUDataclass containing a list of CommitOperation to commit at once. A [`MultiCommitStep`] is one atomic part of a [`MultiCommitStrategy`]. Each step is identified by its own deterministic ID based on the list of commit operations (hexadecimal sha256). ID is persistent between re-runs if the list of commits is kept the same. rFinitid completedNr cCst|jdkrtdt}|jD]x}t|trZ|d||j||j jq"t|t r|d||j|t |j q"t q"||_dS)Nrz?A MultiCommitStep must have at least 1 commit operation, got 0.sADDsDELETE)r#r ValueErrorrr!r updateZ path_in_repoencoder$rstr is_folderNotImplementedError hexdigestr.)selfshar)rrr __post_init__s     zMultiCommitStep.__post_init__c Csdd|jD}dd|jD}dd|jD}t|dkr|d|jrHdndd t|d ttd d |Dd |jd Sd|jrdnddt|dt|d|jd SdS)zFormat a step for PR description. Formatting can be changed in the future as long as it is single line, starts with `- [ ]`/`- [x]` and contains `self.id`. Must be able to match `STEP_ID_REGEX`. cSsg|]}t|tr|qSr)r!r .0r)rrr s z+MultiCommitStep.__str__..cSs g|]}t|tr|js|qSrr!rr5r;rrrr=s cSs g|]}t|tr|jr|qSrr>r;rrrr=s rz- [x z ] Upload z file(s) totalling css|]}|jjVqdSN)r$r%)r<addrrr sz*MultiCommitStep.__str__..z ()z ] Delete z file(s) and z folder(s) (N)rr#r/rsumr.)r8r(Zfile_deletionsZfolder_deletionsrrr__str__s >0zMultiCommitStep.__str__)rrrrrr r r__annotations__rr.r4r/boolr:rFrrrrr+s  r+c@s`eZdZUdZeeed<eeed<eddZe ed<eddZ e e ed<dd d d Z dS) MultiCommitStrategya Dataclass containing a list of [`MultiCommitStep`] to commit iteratively. A strategy is identified by its own deterministic ID based on the list of its steps (hexadecimal sha256). ID is persistent between re-runs if the list of commits is kept the same. r&r'Fr,r. all_stepsNr0cCsdd|j|jD|_t|jt|jt|jkr>tdt|jdkrTtdt}|j|jD]"}|d||jqf| |_dS)NcSsh|] }|jqSr)r.)r<steprrr sz4MultiCommitStrategy.__post_init__..zIGot duplicate commits in MultiCommitStrategy. All commits must be unique.rz9A MultiCommitStrategy must have at least 1 commit, got 0.znew step) r&r'rJr#r1rr2r3r.r7)r8r9rKrrrr:sz!MultiCommitStrategy.__post_init__) rrrrrr+rGrr.r4rJr r:rrrrrIs   rIr)apirepo_idcommit_messagecommit_descriptionstrategytoken repo_typer cCs.|j|d|d|jdt|||d||dS)Nz[WIP] z (multi-commit rDrOrPrQ)rNtitle descriptionrRrS)Zcreate_pull_requestr.multi_commit_generate_comment)rMrNrOrPrQrRrSrrr multi_commit_create_pull_request s rX)rOrPrQr c Cs0tj||p d|jddd|j|jDdS)N css|]}t|VqdSrA)r4)r<commitrrrrC*sz0multi_commit_generate_comment..)rOrPZmulti_commit_idZmulti_commit_strategy)$MULTI_COMMIT_PR_DESCRIPTION_TEMPLATEformatr.joinr'r&rTrrrrW!s  rW)rVr cCsddt|DS)NcSsh|] }|dqS)r r)r<matchrrrrL1sz4multi_commit_parse_pr_description..) STEP_ID_REGEXfindall)rVrrr!multi_commit_parse_pr_description0srb)rr),rre dataclassesrrhashlibrtypingrrrrr r r Z _commit_apir rZ communityrutilsrZutils._cache_managerrZhf_apir Exceptionrr\Z+MULTI_COMMIT_PR_COMPLETION_COMMENT_TEMPLATEZ(MULTI_COMMIT_PR_CLOSING_COMMENT_TEMPLATEZ9MULTI_COMMIT_PR_CLOSE_COMMENT_FAILURE_NO_CHANGES_TEMPLATEZ:MULTI_COMMIT_PR_CLOSE_COMMENT_FAILURE_BAD_REQUEST_TEMPLATEcompile MULTILINEr`intr*r+rIr4rXrWrbrrrrsZ $    j6