Allow numbers in build names

[huddlej]

This is a second attempt to add support for numbers in build names after #522 unexpectedly broke the Nextstrain team's builds. This PR reduces the constraints on build names for most builds and adds rule-specific wildcard constraints to Nextstrain-dated builds. These constraints prevent dates from being in Nextstrain build names so the standard Nextstrain builds can be saved as dated copies. To prevent ambiguity between the dated JSONs rule and the main workflow's finalize rule, this commit also specifies a preferred rule order for the Nextstrain builds.

(Updated on November 24, 2021)
I tested this new code with the following build config (builds.yaml):

inputs:
  - name: example-data
    metadata: data/example_metadata.tsv
    sequences: data/example_sequences.fasta.gz

default_build_name: my-build-2021-11-24

And then the following command:

snakemake --configfile builds.yaml -p -j 1

I also confirmed that the Nextstrain profile's dryrun worked as expected:

snakemake --profile nextstrain_profiles/nextstrain-gisaid all_regions -np

These changes should allow people to name their builds nearly anything they like including dates (e.g., global-2020-12-22) without breaking the existing Nextstrain naming scheme for dated builds.

• Tested with CI
• Tested with full Nextstrain open build

Resolves #482

[rneher]

@huddlej -- I still get this error here:

\:> snakemake --profile nextstrain_profiles/nextstrain all_regions -n
Building DAG of jobs...
InputFunctionException in line 963 of /home/richard/Projects/nextstrain/ncov/workflow/snakemake_rules/main_workflow.smk:
KeyError: 'global_2021-01-10'
Wildcards:
build_name=global_2021-01-10

[jacaravas]

Is there a reason this hasn't been merged yet? Or did it just fall through the cracks? I have had no issues using the code
build_name = r'(?:[_a-zA-Z0-9-](?!(tip-frequencies)))+',
in the Snakefile "wildcard_constraints" for the last several months.

[emmahodcroft]

I think this works fine for most people's builds, however, in the 'official' Nextstrain builds we also generate dated files - I think this is where the error comes in. (Unless something changed since Jan)

[trvrb]

I was just testing out a ncov/gisaid/global/6m build and ran into this issue. I'd suggest bumping priority of addressing this.

[jameshadfield]

ce0a763 is some good detective work @huddlej!

I've tested locally using snakemake 7.3.7 and can build a DAG without issue for the core nextstrain builds, however when changing the example profile to use a numeric build name I get an error:

## my_profiles/example/builds.yaml changed to include
builds:
    north-america_usa_washington_king-county_123: # name of the build; this can be anything
    
# error
Failed validating 'pattern' in schema['properties']['builds']['propertyNames']:
    OrderedDict([('pattern', '^(?:[_a-zA-Z-](?!(tip-frequencies)))+$')])

Updating workflow/schemas/config.schema.yaml lines 39-46 to be the following allowed things to work as expected:

  builds:
    type: object
    minProperties: 1
    propertyNames:
      # Allow build names to contain alpha characters, underscores, and hyphens
      # but not special strings used for Nextstrain builds.  Also used in the
      # workflow's wildcard_constraints.
      pattern: "^(?:[a-zA-Z0-9-_](?!(tip-frequencies)))+$"

[huddlej]

Thank you for testing this out, @jameshadfield! I've rebased onto master (to fix conflicts in the main workflow) and pushed commits to fix the schema validation and update the docs. I confirmed I could build DAGs for Nextstrain GISAID and open builds and have queued up a trial run for open builds.

I also modified the getting_started profile's builds.yaml to name the build global_6m and confirmed that this built as expected with the both CLI/Docker and Snakemake/Conda. If the open builds work as expected, I'll plan to merge this.