Status update for 2020-10-22

[TomAugspurger]

I pushed on pangeo-forge and staged-recipes a bit today. Here's the summary

• Refactored storage / environments (Update tasks for noaa-oisst-avhrr pipeline #16, Handle environment / storage in Pipeline #17).

• Created an example pipeline: Added example recipe pipeline staged-recipes#13, based off the noaa example.
  Note this is just the pipeline definition. I haven't touched the meta.yaml yet.

• Added GitHub Actions for

  1. Linting new recipes. When users make a PR the pipeline is linted (according to
     things like flake8 and pangeo-forge's checks (https://github.com/pangeo-forge/staged-recipes/runs/1294822502, https://github.com/pangeo-forge/staged-recipes/runs/1294822501))

  2. Creating a new repository when a new pipeline is merged to staged-recipes.

     • workflow-file: https://github.com/pangeo-forge/staged-recipes/blob/master/.github/workflows/scripts/add_new_recipes.py
     • sample run: https://github.com/pangeo-forge/staged-recipes/runs/1294959163?check_suite_focus=true
     • sample repo: https://github.com/pangeo-forge/example-pipeline/

• Upon creation of a new repository, it is automatically registered with Prefect via another GitHub action in the new repository.

  • workflow-file: https://github.com/pangeo-forge/example-pipeline/blob/master/.github/workflows/register_pipeline.yaml
  • sample run: https://github.com/pangeo-forge/example-pipeline/runs/1295009345 (slight issue with the original, so that was manually triggered)

• Prefect sees the new flow and runs it

Next step: docs, and then I think we're ready to push on a batch of recipes. I think we'll still have some architectural changes, but hopefully they'll be relatively minor and easy to apply across multiple repositories.

[rabernat]

@TomAugspurger - where is the agent running in the setup you developed this fall?

I think I understand how things work up to the point of running the flow from the workflow. But at that point things hand off to prefect cloud and then the agent picks it up and I lose the thread.

[TomAugspurger]

Right now it's running in the staging namespace on the pangeo-uscentral1b kubernetes cluster.

$ kubectl -n staging get pod -l app=prefect-agent
NAME                             READY   STATUS    RESTARTS   AGE
prefect-agent-58ffbd6d4b-zm7m6   1/1     Running   1          11d

Might make sense to move this to its own namespace at a minimum, possibly its own cluster (is that GCP project going to be executing pangeo-forge pipelines? Other bakeries will necessarily have their own kubernetes clusters running prefect agents)

[TomAugspurger]

I thought I documented the agent setup properly somewhere, but pangeo-forge/terraclimate-feedstock#3 (comment) is possibly the best reference right now.

[TomAugspurger]

I think all the information here is stale now.