Adding recipe for GODAS #21

jbusecke · 2023-05-12T21:35:11Z

Fill out meta.yaml
[ ]

jbusecke · 2023-05-17T00:23:55Z

Well I hadnt seen this before... @cisaacstern

jbusecke · 2023-06-05T21:41:18Z

I almost got the example to run, but I am getting this error:

ValueError: Cannot combine fragments. Expected a hypercube of shape [3] but got 6 fragments. [while running 'Create|OpenURLWithFSSpec|OpenWithXarray|Preprocess|StoreToZarr/StoreToZarr/Rechunk/MapTuple(combine_fragments)-ptransform-31']

indicating to me that this is still not properly concatenting/merging the individual datasets? @cisaacstern does this look familiar to you?

cisaacstern · 2023-06-07T19:32:36Z

@jbusecke apologies for missing this notification. Yes, this seems to be the same issue as pangeo-forge/pangeo-forge-recipes#517, which I am working on a fix for in pangeo-forge/pangeo-forge-recipes#521.

Let's remember to retry this once that fix goes in.

cisaacstern · 2023-06-07T19:37:01Z

Also, the rate limit error above is presumably due to these lines in the action:

https://github.com/pangeo-forge/deploy-recipe-action/blob/bda89250ed9ee256af1d0cac3e951c772ce7b53b/action/deploy_recipe.py#L58-L60

We could resolve this issue if there was a way to rework the action so that it does not need to call back into the GitHub API, and instead relies only on injected environment variables for the necessary context. As of this moment, I'm not sure if/how we might accomplish that. I'll open an issue on the action for that.

feedstock/requirements.txt

Co-authored-by: Charles Stern <62192187+cisaacstern@users.noreply.github.com>

jbusecke · 2023-06-30T19:55:05Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

jbusecke · 2023-06-30T20:06:24Z

@cisaacstern if you have a minute, could you look into this error:

Traceback (most recent call last):
[59](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:60)
  File "/srv/conda/envs/notebook/bin/pangeo-forge-runner", line 5, in <module>
[60](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:61)
    from pangeo_forge_runner.cli import main
[61](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:62)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_runner/cli.py", line 3, in <module>
[62](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:63)
    from .commands.bake import Bake
[63](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:64)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_runner/commands/bake.py", line 15, in <module>
[64](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:65)
    from ..storage import InputCacheStorage, MetadataCacheStorage, TargetStorage
[65](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:66)
  File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_runner/storage.py", line 2, in <module>
[66](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:67)
    from pangeo_forge_recipes.storage import CacheFSSpecTarget, FSSpecTarget, MetadataTarget
[67](https://github.com/leap-stc/data-management/actions/runs/5426518167/jobs/9868667430?pr=21#step:5:68)
ImportError: cannot import name 'MetadataTarget' from 'pangeo_forge_recipes.storage' (/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/storage.py)

Wondering if this is related to pangeo-forge/pangeo-forge-recipes#527 and if there is an easy way for me to circumvent this? Maybe just getting pgf-recipes from the beam-refactor branch is the solution?

cisaacstern · 2023-06-30T20:25:41Z

Wondering if this is related to pangeo-forge/pangeo-forge-recipes#527 and if there is an easy way for me to circumvent this? Maybe just getting pgf-recipes from the beam-refactor branch is the solution?

Yes that's correct. And it also reflects that we should have a better plan for integration testing of recipes + runner.

I'll take a closer look at a fix this afternoon. Thanks for raising this, Julius!

cisaacstern · 2023-06-30T23:31:27Z

Opened pangeo-forge/pangeo-forge-runner#76 to track the issue above. Turns out I won't have time to work up a PR today, but can look at it next week.

cisaacstern · 2023-07-05T03:10:47Z

@jbusecke, following merge of pangeo-forge/pangeo-forge-runner#77, the issue above should be fixed. Can you try re-deploying the test? (I just checked if I could, but I think I don't have permissions to do so on this repo.)

jbusecke · 2023-07-05T15:53:11Z

Rerunning now. Let me also make a dev team here and give you permissions.

jbusecke · 2023-07-05T15:57:13Z

Yay! The deployment worked. Could you try to rerun the test, just to confirm that the 'maintain' role (via new data-management-devs team) has enough permissions to do so?

cisaacstern · 2023-07-05T18:59:14Z

Yep! I can now add/remove labels! Thanks Julius!

jbusecke · 2023-07-05T21:14:27Z

So both runs succeed!

Let me try to add all the variables now.

jbusecke · 2023-07-05T21:19:44Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

jbusecke · 2023-07-05T21:21:54Z

pre-commit.ci autofix

for more information, see https://pre-commit.ci

cisaacstern · 2023-07-05T23:28:44Z

Looking promising! 😄

jbusecke · 2023-07-06T14:45:05Z

Deployment was successful, but there is an issue with combining the surface and depth resolved datasets. Debugging that rn.

jbusecke · 2023-07-06T14:55:47Z

Hmm this looks like some sort of bug in the recipe to me.

This particular dataset contains surface variables like this:

and full 3d variables like this:

When merging this I would expect it to just work, and in the manual case it does:

import fsspec
import xarray as xr

def make_full_path(variable, time):
    return f'https://downloads.psl.noaa.gov/Datasets/godas/{variable}.{time}.nc'

with fsspec.open(make_full_path('vcur', 1980)) as f:
    with fsspec.open(make_full_path('sshg', 1980)) as f_surf:
        ds = xr.open_mfdataset([f, f_surf])
ds

gives

but from the dataflow logs I am getting an error message like this:

apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive File "apache_beam/runners/worker/operations.py", line 1238, in apache_beam.runners.worker.operations.PGBKCVOperation.process File "apache_beam/runners/worker/operations.py", line 1267, in apache_beam.runners.worker.operations.PGBKCVOperation.process File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/combiners.py", line 33, in add_input accumulator.add_input(schema, position) File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/aggregation.py", line 70, in add_input self.schema = _combine_xarray_schemas(self.schema, s, concat_dim=self.concat_dim) File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/aggregation.py", line 82, in _combine_xarray_schemas dims = _combine_dims(s1["dims"], s2["dims"], concat_dim) File "/srv/conda/envs/notebook/lib/python3.9/site-packages/pangeo_forge_recipes/aggregation.py", line 109, in _combine_dims raise DatasetCombineError(f"Dimensions for {dim} have different sizes: {l1}, {l2}") pangeo_forge_recipes.aggregation.DatasetCombineError: Dimensions for level have different sizes: 40, 0 [while running 'Create|OpenURLWithFSSpec|OpenWithXarray|Preprocess|StoreToZarr/StoreToZarr/DetermineSchema/CombineGlobally(CombineXarraySchemas)/KeyWithVoid-ptransform-48']

This indicates to me that somwhere in the aggregation code the surface variables either get promoted to have a scalar level dimension, or that the aggregation can simply not handle this case yet.

@cisaacstern I think we should add this sort of pattern to the PGF-recipes tests?

jbusecke · 2024-09-03T19:40:01Z

This is a leftover from our old mono-feedstock. Can we migrate this into a separate feedstock?
cc @leap-stc/data-and-compute

norlandrhagen · 2024-09-19T17:02:04Z

https://github.com/leap-stc/GODAS_feedstock

Add godas test

ad13dd0

jbusecke added the run:GODAS label May 12, 2023

jbusecke added 6 commits May 12, 2023 17:38

Fix typo in make_full_path

3353df4

switch from climatology to monthly data + add time concat

1fc4f82

Implement Preprocessor

c461a3e

fixed typo

a6f2dc6

add filetype

7634da0

typo

5458926

jbusecke removed the run:GODAS label May 17, 2023

jbusecke added 2 commits May 16, 2023 20:18

Merge branch 'main' into add_godas

201b1ec

try to update fsspec

2011156

jbusecke added the run:GODAS label May 17, 2023

jbusecke added 3 commits June 5, 2023 16:37

Merge branch 'main' into add_godas

c3c6507

Fix double return and typo

8310702

Refactor years name to time

24a17c5

cisaacstern mentioned this pull request Jun 7, 2023

Rate limiting on unauthenticated call to GitHub API pangeo-forge/deploy-recipe-action#9

Open

cisaacstern reviewed Jun 29, 2023

View reviewed changes

feedstock/requirements.txt Outdated Show resolved Hide resolved

cisaacstern mentioned this pull request Jun 29, 2023

Include merge dim positions in group keys emitted by split_fragments pangeo-forge/pangeo-forge-recipes#521

Merged

jbusecke and others added 3 commits June 30, 2023 11:21

Update feedstock/requirements.txt

69cb7d7

Co-authored-by: Charles Stern <62192187+cisaacstern@users.noreply.github.com>

Merge branch 'main' into add_godas

2b95401

run deploy from stderr pr

d6cb4c9

[pre-commit.ci] auto fixes from pre-commit.com hooks

091cc24

for more information, see https://pre-commit.ci

cisaacstern mentioned this pull request Jun 30, 2023

Make import of MetadataTarget conditional on recipes version (or duck typed equivalent) pangeo-forge/pangeo-forge-runner#76

Closed

cisaacstern added run:GODAS and removed run:GODAS labels Jul 5, 2023

cisaacstern mentioned this pull request Jul 5, 2023

Release + testing integration with -runner & action pangeo-forge/pangeo-forge-recipes#545

Open

Merge branch 'main' into add_godas

f9909a7

Add all variables

2db546c

pre-commit-ci bot and others added 2 commits July 5, 2023 21:19

[pre-commit.ci] auto fixes from pre-commit.com hooks

434ac64

for more information, see https://pre-commit.ci

add all years

6ca8b38

jbusecke marked this pull request as ready for review July 5, 2023 21:22

[pre-commit.ci] auto fixes from pre-commit.com hooks

2174430

for more information, see https://pre-commit.ci

jbusecke added 3 commits July 26, 2023 18:16

Update deploy.yaml

090d4c4

Update requirements.txt

b418a1b

Update requirements.txt

c2aecd6

jbusecke added dataset and removed run:GODAS labels Sep 3, 2024

norlandrhagen closed this Sep 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding recipe for GODAS #21

Adding recipe for GODAS #21

jbusecke commented May 12, 2023

jbusecke commented May 17, 2023

jbusecke commented Jun 5, 2023

cisaacstern commented Jun 7, 2023 •

edited

Loading

cisaacstern commented Jun 7, 2023

jbusecke commented Jun 30, 2023

jbusecke commented Jun 30, 2023 •

edited

Loading

cisaacstern commented Jun 30, 2023

cisaacstern commented Jun 30, 2023

cisaacstern commented Jul 5, 2023

jbusecke commented Jul 5, 2023

jbusecke commented Jul 5, 2023

cisaacstern commented Jul 5, 2023

jbusecke commented Jul 5, 2023

jbusecke commented Jul 5, 2023

jbusecke commented Jul 5, 2023

cisaacstern commented Jul 5, 2023

jbusecke commented Jul 6, 2023

jbusecke commented Jul 6, 2023

jbusecke commented Sep 3, 2024

norlandrhagen commented Sep 19, 2024

Adding recipe for GODAS #21

Adding recipe for GODAS #21

Conversation

jbusecke commented May 12, 2023

jbusecke commented May 17, 2023

jbusecke commented Jun 5, 2023

cisaacstern commented Jun 7, 2023 • edited Loading

cisaacstern commented Jun 7, 2023

jbusecke commented Jun 30, 2023

jbusecke commented Jun 30, 2023 • edited Loading

cisaacstern commented Jun 30, 2023

cisaacstern commented Jun 30, 2023

cisaacstern commented Jul 5, 2023

jbusecke commented Jul 5, 2023

jbusecke commented Jul 5, 2023

cisaacstern commented Jul 5, 2023

jbusecke commented Jul 5, 2023

jbusecke commented Jul 5, 2023

jbusecke commented Jul 5, 2023

cisaacstern commented Jul 5, 2023

jbusecke commented Jul 6, 2023

jbusecke commented Jul 6, 2023

jbusecke commented Sep 3, 2024

norlandrhagen commented Sep 19, 2024

cisaacstern commented Jun 7, 2023 •

edited

Loading

jbusecke commented Jun 30, 2023 •

edited

Loading