U -e@sdZddlZddlZddlZddlZddlZddlmZzddlmZWne k r`dZYnXz ddl Z Wne k rdZ YnXdddZ Gd d d Z Gd d d Z Gd ddZGdddZddZddZGdddZGdddeZGdddeZGdddeZdS)z1 The classes that actually handle the downloads. N parse_url)tqdmFcCs^tttttd}t|}|d|krHtd|dd|d|d||d|d}|S)a Choose the appropriate downloader for the given URL based on the protocol. Parameters ---------- url : str A URL (including protocol). progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. Returns ------- downloader A downloader class, like :class:`pooch.HTTPDownloader`, :class:`pooch.FTPDownloader`, or :class: `pooch.SFTPDownloader`. Examples -------- >>> downloader = choose_downloader("http://something.com") >>> print(downloader.__class__.__name__) HTTPDownloader >>> downloader = choose_downloader("https://something.com") >>> print(downloader.__class__.__name__) HTTPDownloader >>> downloader = choose_downloader("ftp://something.com") >>> print(downloader.__class__.__name__) FTPDownloader >>> downloader = choose_downloader("doi:DOI/filename.csv") >>> print(downloader.__class__.__name__) DOIDownloader )ftphttpshttpsftpdoiprotocolzUnrecognized URL protocol 'z' in 'z'. Must be one of .) progressbar) FTPDownloaderHTTPDownloaderSFTPDownloader DOIDownloaderr ValueErrorkeys)urlr Zknown_downloaders parsed_url downloaderrR/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/pooch/downloaders.pychoose_downloaders& rc@s$eZdZdZd ddZd ddZdS) ra Download manager for fetching files over HTTP/HTTPS. When called, downloads the given file URL into the specified local file. Uses the :mod:`requests` library to manage downloads. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to customize the download of files (for example, to use authentication or print a progress bar). Parameters ---------- progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. chunk_size : int Files are streamed *chunk_size* bytes at a time instead of loading everything into memory at one. Usually doesn't need to be changed. **kwargs All keyword arguments given when creating an instance of this class will be passed to :func:`requests.get`. Examples -------- Download one of the data files from the Pooch repository: >>> import os >>> from pooch import __version__, check_version >>> url = "https://github.com/fatiando/pooch/raw/{}/data/tiny-data.txt" >>> url = url.format(check_version(__version__, fallback="main")) >>> downloader = HTTPDownloader() >>> # Not using with Pooch.fetch so no need to pass an instance of Pooch >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> os.path.exists("tiny-data.txt") True >>> with open("tiny-data.txt") as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove("tiny-data.txt") Authentication can be handled by passing a user name and password to :func:`requests.get`. All arguments provided when creating an instance of the class are forwarded to :func:`requests.get`. We'll use ``auth=(username, password)`` to use basic HTTPS authentication. The https://httpbin.org website allows us to make a fake a login request using whatever username and password we provide to it: >>> user = "doggo" >>> password = "goodboy" >>> # httpbin will ask for the user and password we provide in the URL >>> url = f"https://httpbin.org/basic-auth/{user}/{password}" >>> # Trying without the login credentials causes an error >>> downloader = HTTPDownloader() >>> try: ... downloader(url=url, output_file="tiny-data.txt", pooch=None) ... except Exception: ... print("There was an error!") There was an error! >>> # Pass in the credentials to HTTPDownloader >>> downloader = HTTPDownloader(auth=(user, password)) >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> with open("tiny-data.txt") as f: ... for line in f: ... print(line.rstrip()) { "authenticated": true, "user": "doggo" } >>> os.remove("tiny-data.txt") FcKs0||_||_||_|jdkr,tdkr,tddSNT2Missing package 'tqdm' required for progress bars.)kwargsr chunk_sizerrselfr rrrrr__init__s zHTTPDownloader.__init__cCs8|r$tj|dd}t|jdk}|S|j}|ddt|d }|rTt|d}ztj |f|}| |j |j d} t|j dd } |jdkrttjd k} t| d | d ddd } n|jr|j} | | _| D],} | r|| ||jr| |j q|jr | | | | W5|r2| XdS)a Download the given URL over HTTP to the given output file. Uses :func:`requests.get`. Parameters ---------- url : str The URL to the file you want to download. output_file : str or file-like object Path (and file name) to which the file will be downloaded. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. check_only : bool If True, will only check if a file exists on the server and **without downloading the file**. Will return ``True`` if the file exists and ``False`` otherwise. Returns ------- availability : bool or None If ``check_only==True``, returns a boolean indicating if the file is available on the server. Otherwise, returns ``None``. T)allow_redirectsstreamwritew+b)rzcontent-lengthrwin32OBtotalZncolsasciiunitZ unit_scaleZleaveN)requestsheadbool status_codercopy setdefaulthasattropenclosegetraise_for_status iter_contentrintheadersr sysplatformrr+r%flushupdatereset)r r output_filepooch check_onlyresponse availablerispathcontentr+ use_asciiprogresschunkrrr__call__sP         zHTTPDownloader.__call__N)Fr)F__name__ __module__ __qualname____doc__r!rKrrrrrUsL rc@s$eZdZdZd dd Zd d d ZdS)ra Download manager for fetching files over FTP. When called, downloads the given file URL into the specified local file. Uses the :mod:`ftplib` module to manage downloads. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to customize the download of files (for example, to use authentication or print a progress bar). Parameters ---------- port : int Port used for the FTP connection. username : str User name used to login to the server. Only needed if the server requires authentication (i.e., no anonymous FTP). password : str Password used to login to the server. Only needed if the server requires authentication (i.e., no anonymous FTP). Use the empty string to indicate no password is required. account : str Some servers also require an "account" name for authentication. timeout : int Timeout in seconds for ftp socket operations, use None to mean no timeout. progressbar : bool If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. **Custom progress bars are not yet supported.** chunk_size : int Files are streamed *chunk_size* bytes at a time instead of loading everything into memory at one. Usually doesn't need to be changed.  anonymousNFrcCsH||_||_||_||_||_||_||_|jdkrDtdkrDtddSr) portusernamepasswordaccounttimeoutr rrr)r rTrUrVrWrXr rrrrr!s zFTPDownloader.__init__c sbt|}tj|jd}|j|d|jd|rztj|d\}}z(|j |j |j |j d|||k} W5| X| Std } | rtdz|j |j |j |j dd|d} |jr.|d ttjd k} tt||dd | d d d d&fdd} |j| | |jdW5QRXn|j| j|jdW5|| r\ XdS)aZ Download the given URL over FTP to the given output file. Parameters ---------- url : str The URL to the file you want to download. output_file : str or file-like object Path (and file name) to which the file will be downloaded. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. check_only : bool If True, will only check if a file exists on the server and **without downloading the file**. Will return ``True`` if the file exists and ``False`` otherwise. Returns ------- availability : bool or None If ``check_only==True``, returns a boolean indicating if the file is available on the server. Otherwise, returns ``None``. )rXnetloc)hostrTpath)userpasswdZacctr%r&zRETR zTYPE Ir'r(r)Tr*cst||dSz+Update the progress bar and write to outputN)r?lenr%)datarArIrrcallbacklsz(FTPDownloader.__call__..callback) blocksizeN)rftplibFTPrXconnectrTosr[splitr6loginrUrVrWZnlstr4r5quitr voidcmdr0r<r=rr:sizeZ retrbinaryrr%)r rrArBrCrr directory file_namerErFcommandrHrbrrarrK2sF     zFTPDownloader.__call__)rQrRrSrSNFr)FrLrrrrrs& rc@s"eZdZdZd ddZd d ZdS) ra Download manager for fetching files over SFTP. When called, downloads the given file URL into the specified local file. Requires `paramiko `__ to be installed. Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to customize the download of files (for example, to use authentication or print a progress bar). Parameters ---------- port : int Port used for the SFTP connection. username : str User name used to login to the server. Only needed if the server requires authentication (i.e., no anonymous SFTP). password : str Password used to login to the server. Only needed if the server requires authentication (i.e., no anonymous SFTP). Use the empty string to indicate no password is required. timeout : int Timeout in seconds for sftp socket operations, use None to mean no timeout. progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. rRrSNFcCsh||_||_||_||_||_||_g}|jr@tdkr@|dtdkrR|d|rdt d |dS)Nrz7Missing package 'paramiko' required for SFTP downloads. ) rTrUrVrWrXr rappendparamikorjoin)r rTrUrVrWrXr errorsrrrr!s   zSFTPDownloader.__init__c st|}tj|d|jfd}d}z|j|j|jdtj |}|j | _ |j rt||dj}ttjdk}t|d|dd d d |j rˆ&fd d } |j|d|| d W5QRXn||d|W5||dk r|XdS)aQ Download the given URL over SFTP to the given output file. The output file must be given as a string (file name/path) and not an open file object! Otherwise, paramiko cannot save to that file. Parameters ---------- url : str The URL to the file you want to download. output_file : str Path (and file name) to which the file will be downloaded. **Cannot be a file object**. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. rY)sockN)rUrVr[r'r(r)Tr*cs"t|_t|jdSr^)r:r+r?n)currentr+rIrrrbs z)SFTPDownloader.__call__..callback)rb)rrs TransportrTr6rfrUrVZ SFTPClientZfrom_transportrXZ get_channel settimeoutr r:statst_sizer0r<r=rr7) r rrArBr connectionr rlrHrbrryrrKs4    zSFTPDownloader.__call__)rprRrSrSNFrLrrrrr{s" rc@s"eZdZdZd ddZddZdS) ra< Download manager for fetching files from Digital Object Identifiers (DOIs). Open-access data repositories often issue Digital Object Identifiers (DOIs) for data which provide a stable link and citation point. The trick is finding out the download URL for a file given the DOI. When called, this downloader uses the repository's public API to find out the download URL from the DOI and file name. It then uses :class:`pooch.HTTPDownloader` to download the URL into the specified local file. Allowing "URL"s to be specified with the DOI instead of the actual HTTP download link. Uses the :mod:`requests` library to manage downloads and interact with the APIs. The **format of the "URL"** is: ``doi:{DOI}/{file name}``. Notice that there are no ``//`` like in HTTP/FTP and you must specify a file name after the DOI (separated by a ``/``). Use with :meth:`pooch.Pooch.fetch` or :func:`pooch.retrieve` to be able to download files given the DOI instead of an HTTP link. Supported repositories: * `figshare `__ * `Zenodo `__ * `DataVerse `__ instances .. attention:: DOIs from other repositories **will not work** since we need to access their particular APIs to find the download links. We welcome suggestions and contributions adding new repositories. Parameters ---------- progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. chunk_size : int Files are streamed *chunk_size* bytes at a time instead of loading everything into memory at one. Usually doesn't need to be changed. **kwargs All keyword arguments given when creating an instance of this class will be passed to :func:`requests.get`. Examples -------- Download one of the data files from the figshare archive of Pooch test data: >>> import os >>> downloader = DOIDownloader() >>> url = "doi:10.6084/m9.figshare.14763051.v1/tiny-data.txt" >>> # Not using with Pooch.fetch so no need to pass an instance of Pooch >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> os.path.exists("tiny-data.txt") True >>> with open("tiny-data.txt") as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove("tiny-data.txt") Same thing but for our Zenodo archive: >>> url = "doi:10.5281/zenodo.4924875/tiny-data.txt" >>> downloader(url=url, output_file="tiny-data.txt", pooch=None) >>> os.path.exists("tiny-data.txt") True >>> with open("tiny-data.txt") as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove("tiny-data.txt") FrcKs||_||_||_dSN)rr rrrrrr!<szDOIDownloader.__init__c Csht|}t|d}|d}|ddkr4|dd}||}tf|j|jd|j}||||dS)a7 Download the given DOI URL over HTTP to the given output file. Uses the repository's API to determine the actual HTTP download URL from the given DOI. Uses :func:`requests.get`. Parameters ---------- url : str The URL to the file you want to download. output_file : str or file-like object Path (and file name) to which the file will be downloaded. pooch : :class:`~pooch.Pooch` The instance of :class:`~pooch.Pooch` that is calling this method. rYr[r/rN)r r)rdoi_to_repository download_urlrr rr) r rrArBrdata_repositoryrnrrrrrrKAs    zDOIDownloader.__call__N)FrrLrrrrrsQ rcCsJtd|}|j}d|jkr,dkrFnntd|d|d|S)z Follow a DOI link to resolve the URL of the archive. Parameters ---------- doi : str The DOI of the archive. Returns ------- url : str The URL of the archive in the data repository. zhttps://doi.org/XzArchive with doi:z not found (see z). Is the DOI correct?)r.r7rr1r)r rDrrrr doi_to_urlfsrcCsv|ddkr|dd}tttg}t|}d}|D]}|dkr2|j||d}q2|dkrrt|d}td|d|S)a] Instantiate a data repository instance from a given DOI. This function implements the chain of responsibility dispatch to the correct data repository class. Parameters ---------- doi : str The DOI of the archive. Returns ------- data_repository : DataRepository The data repository object rN) archive_urlr rYzInvalid data repository 'zy'. To request or contribute support for this repository, please open an issue at https://github.com/fatiando/pooch/issues)FigshareRepositoryZenodoRepositoryDataverseRepositoryr initializerr)r Z repositoriesrrrepo repositoryrrrrs(    rc@s(eZdZeddZddZddZdS)DataRepositorycCsdS) Initialize the data repository if the given URL points to a corresponding repository. Initializes a data repository object. This is done as part of a chain of responsibility. If the class cannot handle the given repository URL, it returns `None`. Otherwise a `DataRepository` instance is returned. Parameters ---------- doi : str The DOI that identifies the repository archive_url : str The resolved URL for the DOI Nr)clsr rrrrrszDataRepository.initializecCstdS)n Use the repository API to get the download URL for a file given the archive URL. Parameters ---------- file_name : str The name of the file in the archive that will be downloaded. Returns ------- download_url : str The HTTP URL that can be used to download the file. NNotImplementedError)r rnrrrrszDataRepository.download_urlcCstdS) Populate the registry using the data repository's API Parameters ---------- pooch : Pooch The pooch instance that the registry will be added to. Nr)r rBrrrpopulate_registrys z DataRepository.populate_registryN)rMrNrO classmethodrrrrrrrrs rc@s<eZdZddZeddZeddZddZd d Z d S) rcCs||_||_d|_dSrrr _api_responser r rrrrr!szZenodoRepository.__init__cCs"t|}|ddkrdS|||S)rrYz zenodo.orgNrrr rZparsed_archive_urlrrrrs zZenodoRepository.initializecCs6|jdkr0|jdd}td||_|jS)zCached API response from ZenodoNrrzhttps://zenodo.org/api/records/)rrrhr.r7json)r article_idrrr api_response s   zZenodoRepository.api_responsecCsPdd|jdD}||kr%sz1ZenodoRepository.download_url..filesFile '' not found in data archive  (doi:).linksr rrrr r rnrrrrrrszZenodoRepository.download_urlcCs&|jdD]}|d|j|d<q dS)rrZchecksumrNrregistryr rBfiledatarrrr-s z"ZenodoRepository.populate_registryN) rMrNrOr!rrpropertyrrrrrrrrs  rc@sDeZdZddZeddZddZeddZd d Z d d Z d S)rcCs||_||_d|_dSrrrrrrr!<szFigshareRepository.__init__cCs"t|}|ddkrdS|||S)rrYz figshare.comNrrrrrrAs zFigshareRepository.initializecCsB|jd\}}|dd}|ddkr.dSt|dd}|S)zi Parse version from the doi Return None if version is not available in the doi. rr rrvNr)r rhr:)r _suffixZ last_partversionrrr_parse_version_from_doi[s  z*FigshareRepository._parse_version_from_doicCs|jdkrtd|jd}|d}|}|dkr^td|jdtd|}nd|d|}t|}| |d |_|jS) z!Cached API response from FigshareNz)https://api.figshare.com/v2/articles?doi=ridzThe Figshare DOI 'zv' doesn't specify which version of the repository should be used. Figshare will point to the latest version available.z%https://api.figshare.com/v2/articles/z /versions/r) rr.r7r rrwarningswarn UserWarningr8)r ZarticlerrZapi_urlrDrrrrks(     zFigshareRepository.api_responsecCsHdd|jD}||kr8td|d|jd|jd||d}|S)rcSsi|]}|d|qS)namerrrrrrsz3FigshareRepository.download_url..rrrrrrrrrrrs zFigshareRepository.download_urlcCs(|jD]}d|d|j|d<qdS)rmd5:Z computed_md5rNrrrrrrs z$FigshareRepository.populate_registryN) rMrNrOr!rrrrrrrrrrrr;s  'rc@sVeZdZddZeddZeddZeddZej d dZd d Z d d Z dS)rcCs||_||_d|_dSrrrrrrr!szDataverseRepository.__init__cCs>|||}d|jkr"dkr*nndS|||}||_|S)rrrN)_get_api_responser1r)rr rrDrrrrrs   zDataverseRepository.initializecCs.t|}t|dd|dd|}|S)z Perform the actual API request This has been separated into a separate ``classmethod``, as it can be used prior and after the initialization. r ://rYz-/api/datasets/:persistentId?persistentId=doi:)rr.r7)rr rparsedrDrrrrs z%DataverseRepository._get_api_responsecCs"|jdkr||j|j|_|jS)z-Cached API response from a DataVerse instanceN)rrr r)r rrrrs  z DataverseRepository.api_responsecCs ||_dS)zUpdate the cached API responseN)r)r rDrrrrscCst|j}|jdddD]<}||ddkr |dd|dd |dd Sq td |d |jd |jddS)rr` latestVersionrdataFilefilenamer rrYz0/api/access/datafile/:persistentId?persistentId=Z persistentIdrrrrN)rrrrrr )r rnrrrrrrs "z DataverseRepository.download_urlcCs@|jdddD]$}d|dd|j|dd<qdS) rr`rrrrmd5rN)rrrrrrrrs  z%DataverseRepository.populate_registryN) rMrNrOr!rrrrrsetterrrrrrrrs    r)F)rPrgr<rdrr.utilsrr ImportErrorrsrrrrrrrrrrrrrrrs6     7%o|55R~