diff --git a/ci/requirements/doc.yml b/ci/requirements/doc.yml index cbbbaa16d7a..d0484c5e999 100644 --- a/ci/requirements/doc.yml +++ b/ci/requirements/doc.yml @@ -8,6 +8,7 @@ dependencies: - bottleneck - cartopy - cfgrib + - kerchunk - dask-core>=2022.1 - dask-expr - hypothesis>=6.75.8 diff --git a/doc/combined.json b/doc/combined.json new file mode 100644 index 00000000000..345462e055f --- /dev/null +++ b/doc/combined.json @@ -0,0 +1,30 @@ +{ + "version": 1, + "refs": { + ".zgroup": "{\"zarr_format\":2}", + "foo/.zarray": "{\"chunks\":[4,5],\"compressor\":null,\"dtype\":\"`_ is a Python library +that allows you to access chunked and compressed data formats (such as NetCDF3, NetCDF4, HDF5, GRIB2, TIFF & FITS), +many of which are primary data formats for many data archives, by viewing the +whole archive as an ephemeral `Zarr`_ dataset which allows for parallel, chunk-specific access. + +Instead of creating a new copy of the dataset in the Zarr spec/format or +downloading the files locally, Kerchunk reads through the data archive and extracts the +byte range and compression information of each chunk and saves as a ``reference``. +These references are then saved as ``json`` files or ``parquet`` (more efficient) +for later use. You can view some of these stored in the `references` +directory `here `_. + + +.. note:: + These references follow this `specification `_. + Packages like `kerchunk`_ and `virtualizarr `_ + help in creating and reading these references. + + +Reading these data archives becomes really easy with ``kerchunk`` in combination +with ``xarray``, especially when these archives are large in size. A single combined +reference can refer to thousands of the original data files present in these archives. +You can view the whole dataset with from this `combined reference` using the above packages. + +The following example shows opening a combined references generated from a ``.hdf`` file stored locally. + +.. ipython:: python + + storage_options = { + "target_protocol": "file", + } + + # add the `remote_protocol` key in `storage_options` if you're accessing a file remotely + + ds1 = xr.open_dataset( + "./combined.json", + engine="kerchunk", + storage_options=storage_options, + ) + + ds1 + +.. note:: + + You can refer to the `project pythia kerchunk cookbook `_ + and the `pangeo guide on kerchunk `_ for more information. + + .. _io.iris: Iris