o \h-@sPddlZddlZddlZddlmZddlmZddlmZGdddeZ dS) N)AbstractFileSysteminfer_storage_options) MemoryFilecseZdZdZdZdZdZdZ d&fdd Ze d d Z e d'd d Z e ddZ e ddZe ddZd(ddZd)ddZe fddZeddZ    d*ddZd(d d!Zd)d"d#Zd$d%ZZS)+GithubFileSystemaInterface to files in github An instance of this class provides the files residing within a remote github repository. You may specify a point in the repos history, by SHA, branch or tag (default is current master). For files less than 1 MB in size, file content is returned directly in a MemoryFile. For larger files, or for files tracked by git-lfs, file content is returned as an HTTPFile wrapping the ``download_url`` provided by the GitHub API. When using fsspec.open, allows URIs of the form: - "github://path/file", in which case you must specify org, repo and may specify sha in the extra args - 'github://org:repo@/precip/catalog.yml', where the org and repo are part of the URI - 'github://org:repo@sha/precip/catalog.yml', where the sha is also included ``sha`` can be the full or abbreviated hex of the commit you want to fetch from, or a branch or tag name (so long as it doesn't contain special characters like "/", "?", which would have to be HTTP-encoded). For authorised access, you must provide username and token, which can be made at https://github.com/settings/tokens z9https://api.github.com/repos/{org}/{repo}/git/trees/{sha}zChttps://api.github.com/repos/{org}/{repo}/contents/{path}?ref={sha}github)<r Nc  stjd i|||_||_|du|duArtd||_||_|dur(||_|durJd}tj |j ||dfd|ji|j } | | d}||_|dzddlm} | d i||_WdStypd|_YdSw) Nz%Auth required both username and tokenz)https://api.github.com/repos/{org}/{repo}orgrepotimeoutdefault_branchr)HTTPFileSystem)super__init__r r ValueErrorusernametokenrrequestsgetformatkwraise_for_statusjsonrootlshttprhttp_fs ImportError) selfr r sharrrkwargsurr __class__r`/home/air/segue/gemini/backup/venv/lib/python3.10/site-packages/fsspec/implementations/github.pyr,s8      zGithubFileSystem.__init__cCs|jr d|j|jfiSiS)Nauth)rrr#rrr*rJszGithubFileSystem.kwTcCs@tjdddg|d|d|jd}|dd|DS) avList repo names for given org or user This may become the top level of the FS Parameters ---------- org_or_user: str Name of the github org or user to query is_org: bool (default True) Whether the name is an organisation (True) or user (False) Returns ------- List of string zhttps://api.github.com/usersorgs/z/repos)rcSg|]}|dqSnamer).0r rrr* fz*GithubFileSystem.repos..)rrrrr)cls org_or_useris_orgr'rrr*reposPs zGithubFileSystem.reposcCFtjd|jd|jdfd|ji|j}|dd|DS)zNames of tags in the repohttps://api.github.com/repos/r/z/tagsrcSr0r1rr3trrr*r4qr5z)GithubFileSystem.tags..rrr r rrrrr#r'rrr*tagshzGithubFileSystem.tagscCr:)zNames of branches in the repor;r/z /branchesrcSr0r1rr<rrr*r4|r5z-GithubFileSystem.branches..r>r?rrr*branchessrAzGithubFileSystem.branchescCs|j|jdS)z#Named references, tags and branchesr@rBrCr,rrr*refs~szGithubFileSystem.refsFc  s|dkr|p |j}|duradd}d|p |j}|D]=}|jd||d}r5d|n|7fdd|D}|sGt|d}|d d kr\|rX|gSS|d }q#|jvsm||jdfvrtj|j j |j |j |d fd |j i|j} | jdkrt| d ddfdd| dD}||jdfvr||j<n|j}|r|Stdd|DS)a List files at given path Parameters ---------- path: str Location to list, relative to repo root detail: bool If True, returns list of dicts, one per file; if False, returns list of full filenames only sha: str (optional) List at the given point in the repo history, branch or tag name or commit SHA _sha: str (optional) List this specific tree object (used internally to descend into trees) rNr/T)r$_shacsg|] }|dkr|qSr1r)r3o)so_farrr*r4sz'GithubFileSystem.ls..rtypefiler$)r r r$r directory)blobtreec sXg|](}|dvrrd|dn|d|d|d|dd|ddqS)rHr/pathmodesizerr$)r2rOrHrPr$)rr3f)rNtypesrr*r4s   rMcSr0r1rrQrrr*r4r5)_strip_protocolrrstripsplitrFileNotFoundErrordircacherrurlrr r rr status_coderrsorted) r#rNdetailr$rEr%partspartoutr'r)rNrGrSr*rsR            zGithubFileSystem.lscCs|jdSN)rXclear)r#rNrrr*invalidate_cachesz!GithubFileSystem.invalidate_cachecs*t|}d|vrt|S|ddS)NrrNr/)rrrTlstrip)r6rNoptsr(rr*rTs z GithubFileSystem._strip_protocolcCs>t|}d|vr iS|d|dd}|dr|d|d<|S)Nrpasswordr hostr$r)rNrdr_rrr*_get_kwargs_from_urlss z&GithubFileSystem._get_kwargs_from_urlsrbc Ks|dkrt|jj|j|j||p|jd}tj|fd|ji|j }|j dkr+t || | } | drIt| d} | dsItdd| S|jdurRtd|jj| df|||d |S) Nrhr r rNr$rrJcontents#version https://git-lfs.github.com/zRPlease install fsspec[http] to access github files >1 MB or git-lfs tracked files. download_url)rO block_size cache_options)NotImplementedError content_urlrr r rrrrrrZrWrrbase64 b64decode startswithrr!r"open) r#rNrOrlrmr$r%rYr' content_jsonrjrrr*_opens6     zGithubFileSystem._opencCs0|j|||d}t|D] }|j||dq dS)N) recursivemaxdepth)message) expand_pathreversedrm_file)r#rNrvrwrxprrr*rm s zGithubFileSystem.rmc Ks$|jstd||}||}|sC|jj|j|j|d|j d}t j |fd|j i|j }|jdkr9t|||d}|jj|j|j||j d}|j }|pYd||d|rbd |ini} t j|f| |j d |j }| d d } td | rd} t| |||dS)aW Remove a file from a specified branch using a given commit message. Since Github DELETE operation requires a branch name, and we can't reliably determine whether the provided SHA refers to a branch, tag, or commit, we assume it's a branch. If it's not, the user will encounter an error when attempting to retrieve the file SHA or delete the file. Parameters ---------- path: str The file's location relative to the repository root. message: str, optional The commit message for the deletion. zAuthentication requiredr/rirrJr$zDelete )rxr$branch)rrrxrzBranch .+ not foundzTRemove only works when the filesystem is initialised from a branch or default (None)N)rrrT_get_sha_from_cacherorr r rcrrrrrrZrWrrdeleteresearchrb) r#rNrxr%r$rYr' delete_urlr~data error_messageerrorrrr*r{s:      zGithubFileSystem.rm_filecCsL|jD]}|D]}|d}|r"||kr"d|vr"|dSq qdS)Nr2r$)rXvaluesr)r#rNentriesentry entry_pathrrr*rGs z$GithubFileSystem._get_sha_from_cache)NNNN)T)FNNr`)rhNNN)__name__ __module__ __qualname____doc__rYroprotocolrrpropertyr classmethodr9r@rBrDrrbrT staticmethodrgrur}r{r __classcell__rrr(r*r s@       B   3 6r) rprrspecrutilsrmemoryrrrrrr*s