-Ph~dZddlZddlZddlZddlmZddlZddlZddlm Z m Z ddl m Z m Z mZmZmZmZmZddlmZmZmZ dd Z dd ZGd dZdZddZdS)z5 The main Pooch class and a factory function for it. N)Path) hash_matches file_hash) check_version get_loggermake_local_storagecache_locationtemporary_fileos_cacheunique_file_name) DOIDownloaderchoose_downloaderdoi_to_repositoryFcf|td}|t|}t|dd}||z }t ||\}} |dvrt |t d| |t||t||}t||||d|`__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. Returns ------- full_path : str The absolute path (including the file name) of the file in the local storage. Examples -------- Download one of the data files from the Pooch repository on GitHub: >>> import os >>> from pooch import __version__, check_version, retrieve >>> # Make a URL for the version of pooch we have installed >>> url = "https://github.com/fatiando/pooch/raw/{}/data/tiny-data.txt" >>> url = url.format(check_version(__version__, fallback="main")) >>> # Download the file and save it locally. Will check the MD5 checksum of >>> # the downloaded file against the given value to make sure it's the >>> # right file. You can use other hashes by specifying different >>> # algorithm names (sha256, sha1, etc). >>> fname = retrieve( ... url, known_hash="md5:70e2afd3fd7e336ae478b1e740a5f08e", ... ) >>> with open(fname) as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> # Running again won't trigger a download and only return the path to >>> # the existing file. >>> fname2 = retrieve( ... url, known_hash="md5:70e2afd3fd7e336ae478b1e740a5f08e", ... ) >>> print(fname2 == fname) True >>> os.remove(fname) Files that are compressed with gzip, xz/lzma, or bzip2 can be automatically decompressed by passing using the :class:`pooch.Decompress` processor: >>> from pooch import Decompress >>> # URLs to a gzip compressed version of the data file. >>> url = ("https://github.com/fatiando/pooch/raw/{}/" ... + "pooch/tests/data/tiny-data.txt.gz") >>> url = url.format(check_version(__version__, fallback="main")) >>> # By default, you would have to decompress the file yourself >>> fname = retrieve( ... url, ... known_hash="md5:8812ba10b6c7778014fdae81b03f9def", ... ) >>> print(os.path.splitext(fname)[1]) .gz >>> # Use the processor to decompress after download automatically and >>> # return the path to the decompressed file instead. >>> fname2 = retrieve( ... url, ... known_hash="md5:8812ba10b6c7778014fdae81b03f9def", ... processor=Decompress(), ... ) >>> print(fname2 == fname) False >>> with open(fname2) as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> os.remove(fname) >>> os.remove(fname2) When downloading archives (zip or tar), it can be useful to unpack them after download to avoid having to do that yourself. Use the processors :class:`pooch.Unzip` or :class:`pooch.Untar` to do this automatically: >>> from pooch import Unzip >>> # URLs to a zip archive with a single data file. >>> url = ("https://github.com/fatiando/pooch/raw/{}/" ... + "pooch/tests/data/tiny-data.zip") >>> url = url.format(check_version(__version__, fallback="main")) >>> # By default, you would get the path to the archive >>> fname = retrieve( ... url, ... known_hash="md5:e9592cb46cf3514a1079051f8a148148", ... ) >>> print(os.path.splitext(fname)[1]) .zip >>> os.remove(fname) >>> # Using the processor, the archive will be unzipped and a list with the >>> # path to every file will be returned instead of a single path. >>> fnames = retrieve( ... url, ... known_hash="md5:e9592cb46cf3514a1079051f8a148148", ... processor=Unzip(), ... ) >>> # There was only a single file in our archive. >>> print(len(fnames)) 1 >>> with open(fnames[0]) as f: ... print(f.read().strip()) # A tiny data file for test purposes only 1 2 3 4 5 6 >>> for f in fnames: ... os.remove(f) Npooch)envversiondownloadupdatez%s data from '%s' to file '%s'. progressbar)rzSHA256 hash of downloaded file: %s Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.) r r r resolvedownload_actionr rinfostrrstream_downloadr) url known_hashfnamepath processor downloaderr full_pathactionverbs J/var/www/html/test/jupyter/venv/lib/python3.11/site-packages/pooch/core.pyretriever)sGn |   } %% $D$ 7 7 7D &I"9j99LFD ''' 4     -    NN      *3KHHHJY JdKKKK   LL  "#i..))    yY666 y>>masterTc j|'t||}||}t|||}t|tr6t j|ddk}| ddz}t||||||} | S)a Create a :class:`~pooch.Pooch` with sensible defaults to fetch data files. If a version string is given, the Pooch will be versioned, meaning that the local storage folder and the base URL depend on the project version. This is necessary if your users have multiple versions of your library installed (using virtual environments) and you updated the data files between versions. Otherwise, every time a user switches environments would trigger a re-download of the data. The version string will be appended to the local storage path (for example, ``~/.mypooch/cache/v0.1``) and inserted into the base URL (for example, ``https://github.com/fatiando/pooch/raw/v0.1/data``). If the version string contains ``+XX.XXXXX``, it will be interpreted as a development version. Does **not** create the local data storage folder. The folder will only be created the first time a download is attempted with :meth:`pooch.Pooch.fetch`. This makes it safe to use this function at the module level (so it's executed on ``import`` and the resulting :class:`~pooch.Pooch` is a global variable). Parameters ---------- path : str, PathLike, list or tuple The path to the local data storage folder. If this is a list or tuple, we'll join the parts with the appropriate separator. The *version* will be appended to the end of this path. Use :func:`pooch.os_cache` for a sensible default. base_url : str Base URL for the remote data source. All requests will be made relative to this URL. The string should have a ``{version}`` formatting mark in it. We will call ``.format(version=version)`` on this string. If the URL does not end in a ``'/'``, a trailing ``'/'`` will be added automatically. version : str or None The version string for your project. Should be PEP440 compatible. If None is given, will not attempt to format *base_url* and no subfolder will be appended to *path*. version_dev : str The name used for the development version of a project. If your data is hosted on Github (and *base_url* is a Github raw link), then ``"master"`` is a good choice (default). Ignored if *version* is None. env : str or None An environment variable that can be used to overwrite *path*. This allows users to control where they want the data to be stored. We'll append *version* to the end of this value as well. registry : dict or None A record of the files that are managed by this Pooch. Keys should be the file names and the values should be their hashes. Only files in the registry can be fetched from the local storage. Files in subdirectories of *path* **must use Unix-style separators** (``'/'``) even on Windows. urls : dict or None Custom URLs for downloading individual files in the registry. A dictionary with the file names as keys and the custom URLs as values. Not all files in *registry* need an entry in *urls*. If a file has an entry in *urls*, the *base_url* will be ignored when downloading it in favor of ``urls[fname]``. retry_if_failed : int Retry a file download the specified number of times if it fails because of a bad connection or a hash mismatch. By default, downloads are only attempted once (``retry_if_failed=0``). Initially, will wait for 1s between retries and then increase the wait time by 1s with each retry until a maximum of 10s. allow_updates : bool or str Whether existing files in local storage that have a hash mismatch with the registry are allowed to update from the remote URL. If a string is passed, we will assume it's the name of an environment variable that will be checked for the true/false value. If ``False``, any mismatch with hashes in the registry will result in an error. Defaults to ``True``. Returns ------- pooch : :class:`~pooch.Pooch` The :class:`~pooch.Pooch` initialized with the given arguments. Examples -------- Create a :class:`~pooch.Pooch` for a release (v0.1): >>> pup = create(path="myproject", ... base_url="http://some.link.com/{version}/", ... version="v0.1", ... registry={"data.txt": "9081wo2eb2gc0u..."}) >>> print(pup.path.parts) # The path is a pathlib.Path ('myproject', 'v0.1') >>> # The local folder is only created when a dataset is first downloaded >>> print(pup.path.exists()) False >>> print(pup.base_url) http://some.link.com/v0.1/ >>> print(pup.registry) {'data.txt': '9081wo2eb2gc0u...'} >>> print(pup.registry_files) ['data.txt'] If this is a development version (12 commits ahead of v0.1), then the ``version_dev`` will be used (defaults to ``"master"``): >>> pup = create(path="myproject", ... base_url="http://some.link.com/{version}/", ... version="v0.1+12.do9iwd") >>> print(pup.path.parts) ('myproject', 'master') >>> print(pup.base_url) http://some.link.com/master/ Versioning is optional (but highly encouraged): >>> pup = create(path="myproject", ... base_url="http://some.link.com/", ... registry={"data.txt": "9081wo2eb2gc0u..."}) >>> print(pup.path.parts) # The path is a pathlib.Path ('myproject',) >>> print(pup.base_url) http://some.link.com/ To place the storage folder at a subdirectory, pass in a list and we'll join the path for you using the appropriate separator for your operating system: >>> pup = create(path=["myproject", "cache", "data"], ... base_url="http://some.link.com/{version}/", ... version="v0.1") >>> print(pup.path.parts) ('myproject', 'cache', 'data', 'v0.1') The user can overwrite the storage path by setting an environment variable: >>> # The variable is not set so we'll use *path* >>> pup = create(path=["myproject", "not_from_env"], ... base_url="http://some.link.com/{version}/", ... version="v0.1", ... env="MYPROJECT_DATA_DIR") >>> print(pup.path.parts) ('myproject', 'not_from_env', 'v0.1') >>> # Set the environment variable and try again >>> import os >>> os.environ["MYPROJECT_DATA_DIR"] = os.path.join("myproject", "env") >>> pup = create(path=["myproject", "not_env"], ... base_url="http://some.link.com/{version}/", ... version="v0.1", ... env="MYPROJECT_DATA_DIR") >>> print(pup.path.parts) ('myproject', 'env', 'v0.1') N)fallback)rtruefalse/)r"base_urlregistryurlsretry_if_failed allow_updates) rformatr isinstancerosenvirongetlowerrstripPooch) r"r1r version_devrr2r3r4r5pups r(creater@s~+>>>??7?33 $W - -D-%%Q }f==CCEEP s##c)H   '#    C Jr*cveZdZdZ ddZedZedZdd Zd Z d Z d Z d Z ddZ dS)r=a Manager for a local data storage that can fetch from a remote source. Avoid creating ``Pooch`` instances directly. Use :func:`pooch.create` instead. Parameters ---------- path : str The path to the local data storage folder. The path must exist in the file system. base_url : str Base URL for the remote data source. All requests will be made relative to this URL. registry : dict or None A record of the files that are managed by this good boy. Keys should be the file names and the values should be their hashes. Only files in the registry can be fetched from the local storage. Files in subdirectories of *path* **must use Unix-style separators** (``'/'``) even on Windows. urls : dict or None Custom URLs for downloading individual files in the registry. A dictionary with the file names as keys and the custom URLs as values. Not all files in *registry* need an entry in *urls*. If a file has an entry in *urls*, the *base_url* will be ignored when downloading it in favor of ``urls[fname]``. retry_if_failed : int Retry a file download the specified number of times if it fails because of a bad connection or a hash mismatch. By default, downloads are only attempted once (``retry_if_failed=0``). Initially, will wait for 1s between retries and then increase the wait time by 1s with each retry until a maximum of 10s. allow_updates : bool Whether existing files in local storage that have a hash mismatch with the registry are allowed to update from the remote URL. If ``False``, any mismatch with hashes in the registry will result in an error. Defaults to ``True``. NrTc||_||_|i}||_|i}t||_||_||_dSN)r"r1r2dictr3r4r5)selfr"r1r2r3r4r5s r(__init__zPooch.__init__sQ    H  <DJJ .*r*c ttjtjt |jS)z"Absolute path to the local storage)rr8r"abspath expanduserrrEs r(rHz Pooch.abspaths8BGOOBG$6$6s49~~$F$FGGHHHr*c*t|jS)z"List of file names on the registry)listr2rJs r(registry_fileszPooch.registry_filessDM"""r*Fc V||||}|j|z }|j|}t ||\}} |dkr|jst |d|d|dvrtt|jt d| ||t|j|t||}t||||||j ||t|||St|S) a* Get the absolute path to a file in the local storage. If it's not in the local storage, it will be downloaded. If the hash of the file in local storage doesn't match the one in the registry, will download a new copy of the file. This is considered a sign that the file was updated in the remote storage. If the hash of the downloaded file still doesn't match the one in the registry, will raise an exception to warn of possible file corruption. Post-processing actions sometimes need to be taken on downloaded files (unzipping, conversion to a more efficient format, etc). If these actions are time or memory consuming, it would be best to do this only once right after the file is downloaded. Use the *processor* argument to specify a function that is executed after the download to perform these actions. See :ref:`processors` for details. Custom file downloaders can be provided through the *downloader* argument. By default, Pooch will determine the download protocol from the URL in the registry. If the server for a given file requires authentication (username and password), use a downloader that support these features. Downloaders can also be used to print custom messages (like a progress bar), etc. See :ref:`downloaders` for details. Parameters ---------- fname : str The file name (relative to the *base_url* of the remote data storage) to fetch from the local storage. processor : None or callable If not None, then a function (or callable object) that will be called before returning the full path and after the file has been downloaded. See :ref:`processors` for details. downloader : None or callable If not None, then a function (or callable object) that will be called to download a given URL to a provided local file name. See :ref:`downloaders` for details. progressbar : bool or an arbitrary progress bar object If True, will print a progress bar of the download to standard error (stderr). Requires `tqdm `__ to be installed. Alternatively, an arbitrary progress bar object can be passed. See :ref:`custom-progressbar` for details. Returns ------- full_path : str The absolute path (including the file name) of the file in the local storage. rz needs to update z but updates are disallowed.rz%s file '%s' from '%s' to '%s'.Nr)rr4)_assert_file_in_registryget_urlrHr2rr5 ValueErrorr rrrrrr4) rEr!r#r$rrr%r r&r's r(fetchz Pooch.fetchs_f %%e,,,ll5!!L5( ]5) &y*==  X  d&8 RR9RRR  + + + s4<00 1 1 1 LL  1DL!!    !.s LLL  $ 4       9S^^VT:: :9~~r*c>||jvrtd|ddS)zg Check if a file is in the registry and raise :class:`ValueError` if it's not. zFile 'z' is not in the registry.N)r2rQrEr!s r(rOzPooch._assert_file_in_registry[s3  % %FeFFFGG G & %r*c|||j|d|j|gS)a Get the full URL to download a file in the registry. Parameters ---------- fname : str The file name (relative to the *base_url* of the remote data storage) to fetch from the local storage. )rOr3r:joinr1rTs r(rPz Pooch.get_urlcsA %%e,,,y}}UBGGT]E,B$C$CDDDr*c tj5}t|dr|}n$|t |d}t |D]\}}t |tr|d}| }| drYtj |}t|dvr,td|d|dzd t|d |d |rQ|d }|d}t|d kr|d} | |j|<||j|< ddddS#1swxYwYdS)a Load entries from a file and add them to the registry. Use this if you are managing many files. Each line of the file should have file name and its hash separated by a space. Hash can specify checksum algorithm using "alg:hash" format. In case no algorithm is provided, SHA256 is used by default. Only one file per line is allowed. Custom download URLs for individual files can be specified as a third element on the line. Line comments can be added and must be prepended with ``#``. Parameters ---------- fname : str | fileobj Path (or open file object) to the registry file. readzutf-8)encoding#)rz&Invalid entry in Pooch registry file 'z$': expected 2 or 3 elements in line rz but got z. Offending entry: ''rr]r\N) contextlib ExitStackhasattr enter_contextopen enumerater7bytesdecodestrip startswithshlexsplitlenOSErrorr3r;r2) rEr!stackfinlinenumlineelements file_name file_checksumfile_urls r( load_registryzPooch.load_registryqs & ! # # Euuf%% I))$uw*G*G*GHH!*3 E E dE**0;;w//Dzz||??3'' ;t,,8}} 11!FFFBFFF E ( I$,QKM8}}))#+A;/7 ),/=E>)NNNNF)Nr+NNNrT)Nr)rr8rr_pathlibrrirhashesrrutilsrrr r r r r downloadersrrrr)r@r=rrrr*r(rs  ,+++++++MLLLLLLLLL  ^^^^H   ttttnmmmmmmmm` ###L)-)-)-)-)-)-r*