Support default GISAID metadata and sequences

[huddlej]

Description of proposed changes

We would like to support GISAID's standard metadata and sequence data formats from the "Download packages" interface of EpiCoV. This PR expands the existing "sanitize" scripts for metadata and sequences with the following major changes to support these data:

• Remove whitespace from strain names in sequence and metadata (whitespace is not allowed in record ids for FASTA deflines)
• Strip metadata from FASTA deflines (these metadata entries should be present already in the corresponding separate metadata file)
• Rename GISAID metadata columns to Augur-style names ("Virus name" -> "strain" and "Collection date" -> "date")
• Parse region, country, division, and location values from the single Location field in the GISAID metadata
• Resolve duplicate metadata records by preferring those with the latest GISAID accession number (with an option to produce an error with a list of all duplicates instead)
• Resolve duplicate sequence records by preferring the first sequence encountered (necessary for downstream components of the workflow when the combine-and-dedup step doesn't get run and duplicates exist in the sequence data)
• Ingest sequences (.fasta) and metadata (.tsv) files directly from GISAID tarballs (allows users to download data and run the workflow directly on those files without manually extracting/decompressing sequences and metadata)

Currently, these changes are applied to all inputs including data that are not in the GISAID default format.

Testing

• Manually test sanitizer scripts locally with the full GISAID downloads
• Run a small build with a 200 random strains from the full downloads
• Tested with a full Nextstrain build on a SLURM cluster
• Tested with AWS Batch trial build

Release checklist

If this pull request introduces backward incompatible changes, complete the following steps for a new release of the workflow:

• Determine the version number for the new release by incrementing the most recent release -> v7
• Update docs/change_log.md in this pull request to document these changes and the new version number.
• After merging, create a new GitHub release with the new version number as the tag and release title.

[emmahodcroft]

Apologies if this is a naive question, is this also stripping out hCoV/ or whatever the header is - so that the root sequence & exclude files will work properly?
Apologies if this is in there and I just missed it while skimming changes 🙃

[huddlej]

@emmahodcroft Yeah, we actually do that as part of the current sanitize scripts based on these config parameters. We had to make that change to get the "Augur input" format downloads to work with our standard include/exclude files and then this PR builds on those existing scripts.

[emmahodcroft]

Thanks John! I had thought I'd seen this somewhere before, but I just couldn't remember exactly where. Apologies for the repeat!

[huddlej]

This PR implements duplicate resolution for metadata using a procedure described as a possible improvement for Augur's own metadata reader. If this approach to resolving duplicates seems reasonable, we can port this function into Augur.

This PR does not implement duplicate resolution for sequences (at least, not yet!).

[huddlej]

After some further modifications to the workflow, we can now run builds like the following that use GISAID tarballs directly as metadata and sequences inputs:

# Define inputs with preferred sequences/metadata listed last.
inputs:
  - name: north-america
    metadata: data/ncov_north-america.tar.gz
    sequences: data/ncov_north-america.tar.gz
  - name: washington
    metadata: data/gisaid_auspice_input_hcov-19_2021_05_24_21.tar
    sequences: data/gisaid_auspice_input_hcov-19_2021_05_24_21.tar

# Define builds.
builds:
  washington:
    region: North America
    country: USA
    division: Washington
    subsampling_scheme: focal-contextual

# Define subsampling scheme.
subsampling:
  focal-contextual:
    focal:
      query: --query "division == '{division}'"
      max_sequences: 20
    contextual:
      query: --query "division != '{division}'"
      max_sequences: 20
      group_by: region year month
      priorities:
        type: proximity
        focus: focal

Internally, the sanitize_sequences.py script extracts the first .tsv or .fasta file from a given (compressed or uncompressed) tarball, decompresses that file as needed, and returns the corresponding buffer to be consumed downstream.

[rneher]

This looks good to me. Thanks John. I left one comment about the pandas.drop_duplicates

[huddlej]

Based on conversation in Slack, I opted to not standardize column names in the metadata in favor of the simpler approach of manually renaming all columns.