Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pathogen-repo-build: Wait for AWS Batch jobs to finish #54

Merged
merged 4 commits into from
Feb 13, 2024

Commits on Feb 6, 2024

  1. dev: Include devel/check-readme in its own checks

    No harm in describing it in the README, esp. as other development
    programs are to be added imminently.
    tsibley committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    db78b7c View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2024

  1. pathogen-repo-build: Generate the workflow YAML from a separately-aut…

    …hored input YAML
    
    This allows us to author the workflow using YAML anchors/references
    (especially with merge keys) since GitHub Actions doesn't otherwise
    support those YAML features.
    
    The lack of support is a shame because GitHub Actions workflows can be
    very repetitive and anchors/references/merges are a decent solution to
    that.  I'm about to add substantial conceptual replication that we won't
    want to maintain concretely replicated in the file.
    
    Since the generated file must be checked in, a new CI step ensures the
    generated file matches the authored file and the generated file is
    excluded from git diffs by default.  An optional local pre-commit hook
    is also available for making sure you craft commits that won't run afoul
    of the CI check later.  Or you can, as necessary, run `make` manually
    before committing.
    tsibley committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    edd3290 View commit details
    Browse the repository at this point in the history
  2. pathogen-repo-build: Support manual triggering with workflow_dispatch

    This lets us more easily test it in development.
    tsibley committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    1f41bea View commit details
    Browse the repository at this point in the history
  3. pathogen-repo-build: Wait for AWS Batch jobs to finish

    This avoids the disconnect between the success/failure status of a
    GitHub Actions workflow run and the actual AWS Batch job, which makes
    for easier reporting and debugging and generally less cognitive
    dissonance.
    
    GitHub Actions workflow _runs_ have a very high max timeout of 35 days,
    but each _job_ in a workflow has a much lower max timeout of 6 hours.
    Many of our builds should be less than 6 hours¹, but here we support
    builds up to 24 hours by chaining together 4 GitHub Actions jobs.  We
    can add more jobs to the chain if we need to, but I don't foresee that.
    
    Nearly all of our builds are in public repos, which means they won't
    consume usage minutes from our GitHub Actions quota.  However, they
    _will_ consume concurrency limits from the quota.  We were already
    frequently bumping into the default free-tier quota of 20 concurrent
    jobs, so adding more long-running jobs (to wait around for the AWS Batch
    jobs) was a nonstarter until we upgraded to a Team plan with its
    corresponding quota of 60 concurrent jobs.²
    
    ¹ As of 24 Jan, all Batch jobs except one in the prior week were sub 12
      hours.  The exception was the GISAID ncov-ingest job launched on 16 Jan.
      Next longest jobs were 10.5 hours, all GenBank ncov-ingest jobs.
      <nextstrain/private#95 (comment)>
    
    ² <https://bedfordlab.slack.com/archives/C7SDVPBLZ/p1705024616778389>
      <https://github.com/tsibley/blab-standup/blob/HEAD/2023-08-22.md>
    tsibley committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    57eb5e5 View commit details
    Browse the repository at this point in the history