wi|BdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZdd lmZd d lmZmZmZmZmZm Z m!Z!m"Z"d d l#m$Z$d d l%m&Z&m'Z'd dl(m)Z)erd dl*m+Z+e j,e-Z.ej/dZ0dZ1dddZ2eGddZ3e" d;dee3dee4de4de4dee4dee4deee4e4fdeee4deee5ee5ee4ffdZ6Gd d!eZ7Gd"d#eZ8 dd.e4ddf d2Z?d0ed3e3d1e>dee4fd4Z@d5eed6e4de8fd7ZAd$d%d8ee4d1e>deefd9ZBd$d%d8ee4d1e>deefd:ZCdS)=z.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize)Path) TYPE_CHECKINGBinaryIODictIterableListOptionalTuple TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_url get_sessionhf_raise_for_status http_backoffloggingtqdmvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)is_tqdm_disabled)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-TypeceZdZUdZeed<eed<eed<edefdZ edefdZ ed e fd Z d S) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesamplepathct|}tj|d5}|ddd}t |}dddn #1swxYwY||||S)Nrbr"rr#)rioopenpeekr)clsr$r"filer#shas k/var/www/development/aibuddy-work/election-extract/venv/lib/python3.11/site-packages/huggingface_hub/lfs.py from_pathzUploadInfo.from_pathOst}} WT4  $DYYs^^DSD)Fd##C $ $ $ $ $ $ $ $ $ $ $ $ $ $ $sS8888s-AA"%A"datact|}|t||dd|S)Nr')r"r#r)rdigestlen)r,r1r.s r/ from_byteszUploadInfo.from_bytesWs>Tll!!##sD $tt*SAAAAfileobjc|d}|dtjt |}|}|dtj||||S)Nr'rr()readseekr)SEEK_SETrtell)r,r7r#r.r"s r/ from_fileobjzUploadInfo.from_fileobj\spc"" Q $$$'""||~~ Q $$$sS8888r6N) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr0r5rr=r6r/r!r!<s   MMM III MMM9S999[9BeBBB[B98999[999r6r! upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturnc||n tj}d}|tjvrtj|}|d||d} d||nddgd|Dd d } |d t|i| d <itt | |pi}t | || } t| | } | dd} t| tstd| d}t|tr|nd}d| Dd| D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcPg|]#}|j|jd$S)oidr")rhexr").0rTs r/ z'post_lfs_batch_info..sG   }((**      r6r) operationrOobjects hash_algonameref)rIrNjsonr^zMalformed response from servertransferc4g|]}d|vt|Serror)_validate_batch_actionsr[objs r/r\z'post_lfs_batch_info..s+OOO#G3.s&III'S.. s # #...r6)rENDPOINTREPO_TYPES_URL_PREFIXESr LFS_HEADERSrrpostrrcget isinstancelist ValueErrorrF)rHrIrJrKrLrMrNrO url_prefix batch_urlpayloadresp batch_infor^chosen_transfers r/post_lfs_batch_infor{fsX$/xxY5GHJI5556yA NNjN'NNNI"+"7YYg{=S  '      G '("3"34   ' ' ' =bG ==  iw  G GDJnnY--G gt $ $;9::: nnZ00O)3OS)I)ISootO POOOOIIwIII r6c$eZdZUeed<eed<dS) PayloadPartT partNumberetagN)r>r?r@rDrCrFrGr6r/r}r}s"OOO IIIIIr6r}c4eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rYpartsN)r>r?r@rArFrCr r}rGr6r/rrs2II HHH  r6rr]rlfs_batch_actionct||d}|%td|jddS|dd}t ||dd}|t ||di}|d} t |d | } | M t| } n(#ttf$rtd | d wxYwt||| | nt|| |t |t |d |} t | t|||jj|jjd} t'| t|jddS)aw Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrTverifyheader chunk_sizehref)rMzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)r]rr upload_url)r]r)rIrNrXrbz: Upload successful)rhrqloggerdebug path_in_repo_validate_lfs_actionrrDrt TypeError_upload_multi_part_upload_single_partrrpr upload_inforrZr"r) r]rrIrNrMr upload_action verify_actionrrr verify_url verify_resps r/ lfs_uploadrs8,---""9--G n (>nnnooo%Y/9M'''$Y/33H==M ]+++  x , ,FL))J' f(=QQQJ ZJJI&   ugquuu   Yv*akllllliJGGGG ]++++M&,A8LL !mm(( $5'BBB".599;;YEZE_``)  K((( LLI*???@@@@@s C**%D lfs_actionct|dtr=|d7t|dtst d|S).validates response from the LFS batch endpointrrNz"lfs_action is improperly formatted)rrrqrFdictrt)rs r/rrse :>>&))3//? ^^H % % -JNN8>> r6lfs_batch_actionsct|dtr(t|dtst d|did}|did}|t ||t ||S)rrYr"z)lfs_batch_actions is improperly formattedrrTr)rrrqrFrDrtr)rrrs r/rhrhs (,,U33S 9 9FjIZI^I^_eIfIfhk>l>lFDEEE%)))R88<.os,    Az    r6cg|]>\}}|r%t|dk,t|d|f?S)r )isdigitr4rD)r[part_numrs r/r\z*_get_sorted_parts_urls..rs\   (Hj##%% +.h--!*;*;Xr""J/*;*;*;r6c|dS)NrrG)ts r/z(_get_sorted_parts_urls..ws !A$r6)keyz0Invalid server response to upload large LFS file)sorteditemsr4rr"rt)rrrsorted_part_upload_urls num_partss r/rrns  #  ,2LLNN           +,,ID)J67777KLLL ""r6rrYcg}t|D]U\}}|d}||dkrtd|d|dz||dz|dV||dS)NrrRzInvalid etag (`z`) returned for part r)r~r)rYr) enumeraterqrtappend)rrYr part_numberrrs r/rrs "E()9::    Vzz&!! <42::[t[[+XY/[[\\ \ )Ao       ' ''r6rcbg}|d5}t|D]m\}}t|||z|5}td||}t |||jdddn #1swxYwYn dddn #1swxYwY|S)NTr) seek_from read_limitrr)rrrrrrrN) r]rrrNr7part_idxpart_upload_url fileobj_slicepart_upload_ress r/rrsKG   T  * * 8g)23D)E)E 8 8 %Ho$x/% 8".uoM"Z"Z"Z#O4446777 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 Ns5*B$ #D-C&%D-& D0DDD--D14D1)NNNN)NNN)DrArr)rer dataclassesrmathros.pathrpathlibrtypingrrr r r r r r urllib.parserhuggingface_hubrutilsrrrrrrrr utils._lfsr utils.sharr utils.tqdmr _commit_apir get_loggerr>rcompile OID_REGEXLFS_MULTIPART_UPLOAD_COMMANDror!rFrr{r}rrrrhrlrrDrrrrrrGr6r/rs\54 !!!!!!\\\\\\\\\\\\\\\\\\\\ %%%%%%                    %$$$$$********((((((0//////  H % % BJ( ) ) 5-2  &9&9&9&9&9&9&9 &9R #"(,%)SS:&S C=SS S sm S sm Sd38n %SS "S 4:tDz8C= 01SSSSl9  (," CACA#CACA C=CAd38n % CA sm CA  CACACACALT t     4    &S&T&&&&, ("6 ( (RU (cf (ko ( ( ( (F#4#j#c#VZ[^V_####$ (d4j (s (GY ( ( ( (#8