o viG7@s|dZddlZddlZddlmZddlmZddlmZddl m Z m Z m Z m Z mZddlmZddlmZd d lmZmZmZmZmZmZd d lmZd d lmZmZe r\d d lm Z e!e"Z#e$dZ%dZ&dddZ'eGdddZ(e    dBde e(de e)de)de)de e)de e)de e*e)e)fde e+e)de,e+e*e+e*e e)ffddZ-Gdd d eZ.Gd!d"d"eZ/   dCd#d$d%e*de e)de e*e)e)fde e)ddf d&d'Z0d(e*fd)d*Z1d+e*fd,d-Z2d.e*fd/d0Z3d#d$d1e)ddfd2d3Z4d#d$d4e*d5e5d1e)ddf d6d7Z6d4e*d8e(d5e5de+e)fd9d:Z7d;e+e*dZ8d#d$d?e+e)d5e5de+e*fd@dAZ9dS)Dz.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize) TYPE_CHECKINGBinaryIOIterableOptional TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_urlhf_raise_for_status http_backoffloggingvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec@s`eZdZUdZeed<eed<eed<edefddZ edefd d Z ed e fd d Z dS) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathcCs\t|}t|d}|ddd}t|}Wdn1s"wY||||dS)Nrbrrr)rioopenpeekr)clsrrfilershar%R/home/air/biblejyuku/back/venv/lib/python3.10/site-packages/huggingface_hub/lfs.py from_pathIs  zUploadInfo.from_pathdatacCs&t|}|t||dd|dS)Nr)rrr)rdigestlen)r"r(r$r%r%r& from_bytesQs zUploadInfo.from_bytesfileobjcCsD|d}|dtjt|}|}|dtj||||dS)Nrrr)readseekrSEEK_SETrtell)r"r,rr$rr%r%r& from_fileobjVs zUploadInfo.from_fileobjN) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr'r+rr1r%r%r%r&r6s  r upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturncCs|dur|ntj}d}|tjvrtj|}|d||d} d|dur&|nddgdd |Dd d } |dur?d t|i| d <itt|d|pJi}td| || d} t| | } | dd} t | t skt d| d}t |t rw|nd}dd | Ddd | D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HfHubHTTPError`] If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcSsg|] }|j|jdqS)oidr)rhexr).0rFr%r%r& s z'post_lfs_batch_info..r) operationrBobjects hash_algonameref)r<POSTrAjsonrOzMalformed response from servertransfercSsg|] }d|vrt|qSerror)_validate_batch_actionsrLobjr%r%r&rMcSsg|] }d|vrt|qSrW)_validate_batch_errorrZr%r%r&rMr\)r ENDPOINTREPO_TYPES_URL_PREFIXESr LFS_HEADERSr rrrUget isinstancelist ValueErrorr:)r;r<r=r>r?r@rArB url_prefix batch_urlpayloadresp batch_inforOchosen_transferr%r%r&post_lfs_batch_info`sB,        rkc@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r2r3r4r8r7r:r%r%r%r&rls  rlc@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rJpartsN)r2r3r4r5r:r7rcrlr%r%r%r&ros rorNrlfs_batch_actionc CsFt||d}|durtd|jddS|dd}t||dd}|dur2t||di}|d} t|d |d } | durjzt| } Wntt fy`td | d wt ||| | d nt || d|durt|t|d |} t d| t ||d|jj|jjdd} t| t|jddS)a9 Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HfHubHTTPError`] If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrFverifyheader chunk_sizehref)r@zTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rNrtru upload_url)rNrwrS)r<rArIrTz: Upload successful)rYraloggerdebug path_in_repo_validate_lfs_actionrr8rd TypeError_upload_multi_part_upload_single_partrr upload_inforrKrr) rNrqr<rAr@rr upload_action verify_actionrtrurw verify_url verify_respr%r%r& lfs_uploadsD        r lfs_actioncCs:t|dtr|ddust|dtstd|S).validates response from the LFS batch endpointrvrtNz"lfs_action is improperly formatted)rbrar:dictrd)rr%r%r&r{ s r{lfs_batch_actionscCspt|dtrt|dtstd|did}|did}|dur.t||dur6t||S)rrJrz)lfs_batch_actions is improperly formattedrrrFrsN)rbrar:r8rdr{)rrrr%r%r&rYs rYlfs_batch_errorcCsht|dtrt|dtstd|d}t|tr.t|dtr.t|dts2td|S)rrJrz'lfs_batch_error is improperly formattedrXmessagecode)rbrar:r8rdr)r error_infor%r%r&r]"s  r]rwcCsH|jdd}td||d}t|WddS1swYdS)aK Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Raises: [`HfHubHTTPError`] If the upload resulted in an error. T with_tqdmPUTr(N)as_filerr)rNrwr,responser%r%r&r~0s "r~rtrucCsHt||j|d}t|||d}td|t||jjtd}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )rtrru)rNsorted_parts_urlsrurS)rUrAN) _get_sorted_parts_urlsr_upload_parts_iterativelyr_get_completion_payloadrrKr`r)rNrtrurwrresponse_headerscompletion_resr%r%r&r}Ds r}rcCsNddtdd|DdddD}t|}|t|j|kr%td|S)NcSsg|]\}}|qSr%r%)rL_rwr%r%r&rM\sz*_get_sorted_parts_urls..cSs2g|]\}}|rt|dkrt|d|fqS)r )isdigitr*r8)rLpart_numrwr%r%r&rM_s  cSs|dS)Nrr%)tr%r%r&dsz(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr*rrrd)rtrrusorted_part_upload_urls num_partsr%r%r&r[s rrrJcCsfg}t|D]'\}}|d}|dus|dkr#td|d|d||d|dq||dS)NrnrDzInvalid etag (`z`) returned for part r )rmrn)rJrp) enumeraterardappend)rrJrp part_numberrtrnr%r%r&rms  rrc Csg}|jdd<}t|D].\}}t||||d}td||d}t|||jWdn1s6wYq Wd|S1sGwY|S)NTr) seek_from read_limitrr)rrrrrrrA) rNrrurAr,part_idxpart_upload_url fileobj_slicepart_upload_resr%r%r&r|s&   r)NNNN)NNN):r5rre dataclassesrmathros.pathrtypingrrrrr urllib.parser huggingface_hubr utilsr rrrrr utils._lfsr utils.sharr _commit_apir get_loggerr2rxcompile OID_REGEXLFS_MULTIPART_UPLOAD_COMMANDr`rr:rrctuplerkrlrorr{rYr]r~r8r}rrrr%r%r%r&s          )  V  G