Original time encoding units not used in Zarr generated by beam-refactor branch

[derekocallaghan]

While porting the EOOffshore CCMP recipe to use the beam-refactor branch, I noticed that the output Zarr contains non-unique time values.

It looks like the original dataset time encoding units aren't retained in pangeo_forge_recipes.writers._store_data():

pangeo-forge-recipes/pangeo_forge_recipes/writers.py

Line 30 in eea8e44

var_coded = var.copy() # copy needed for test suit to avoid modifying inputs in-place

At this point the original time encoding is as expected (i.e. hours...):

{'chunksizes': (524288,), 'fletcher32': False, 'shuffle': False, 'source': '<fsspec.implementations.local.LocalFileOpener object at 0x7f88c259aac0>', 'original_shape': (4,), 'dtype': dtype('float64'), 'units': 'hours since 1987-01-01 00:00:00'}

The units are subsequently replaced after this statement, where the use of days... is resulting in the non-unique values:

pangeo-forge-recipes/pangeo_forge_recipes/writers.py

Line 31 in eea8e44

var_coded.encoding.update(zarr_array.attrs)

{'chunksizes': (524288,), 'fletcher32': False, 'shuffle': False, 'source': '<fsspec.implementations.local.LocalFileOpener object at 0x7f88c259aac0>', 'original_shape': (4,), 'dtype': dtype('float64'), 'units': 'days since 1970-01-01 00:00:00', '_ARRAY_DIMENSIONS': ['time'], '_CoordinateAxisType': 'Time', '_Fillvalue': -9999.0, 'axis': 'T', 'calendar': 'proleptic_gregorian', 'delta_t': '0000-00-00 06:00:00', 'long_name': 'Time of analysis', 'standard_name': 'time'}

I don't have a fix for this yet, but it looks like the issue is in pangeo_forge_recipes.aggregation._to_variable() based on the existing comments:

pangeo-forge-recipes/pangeo_forge_recipes/aggregation.py

Lines 198 to 203 in eea8e44

	# WARNING: there are lots of edge cases aroudn time!
	# Xarray will pick a time encoding for the dataset (e.g. "days since days since 1970-01-01")
	# and this may not be compatible with the actual values in the time coordinate
	# (which we don't know yet)
	data = dsa.zeros(shape=shape, chunks=chunks, dtype=dtype)
	# TODO: add more encoding

[rabernat]

This is such a useful bug report. Thank you so much! 🙏

[derekocallaghan]

I have a potential fix for this almost working locally, I need to resolve some test failures. I'm hoping to create a PR later this week.

[rabernat]

Amazing! Yes, PR very welcome!

[derekocallaghan]

Confirmed that this is resolved by #471.