Virtualizarr + Xarray-beam as a ARCO dataset creation option

[norlandrhagen]

I created an example repo to explore how you can use virtualizarr + xarray-beam to create an ARCO dataset from a collection of NetCDF files.

In this example, I used a pre-generated virtualizarr reference as the input dataset. This was then fed into an apache-beam pipeline using xarray-beam PTransforms to open the virtualizarr, rechunk and materialize it as a Zarr store.

Rough and optimized timings on dataflow.

1.3 TB dataset
13 variables (merge op)
60 years of daily data
33 minutes on dataflow using ARM instances.

ToDo:

• Clean up repo.
• Create stage to create virtualizarr references
• Compare (a subset) against standard pangeo-forge recipe.

cc @jbusecke @SammyAgrawal