WIP: Use metadata-only filtering in subsample jobs #571
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Augur filter's new metadata-only interface will allow users to pass multiple inputs to the `--exclude` argument, internally deduplicating these strain lists. This new interface eliminates the need for a separate Snakemake rule to cat the exclusion files.
Replaces FASTA outputs with strain list outputs for the subsample rule such that sequence data are not inspected during most subsampling steps. The exception to the rule are subsampling jobs that require a priority score calculation that depends on the FASTA sequence of another subsampled group. To handle this exception, we add a new rule to extract just those subsampled sequences. Finally, we collect subsampled sequences into a single deduplicated FASTA output using augur filter's new interface with the `--exclude-all` flag and multiple input support for `--include`. Note that this commit also updates the conda environment to use a GitHub branch instead of an official augur release.
huddlej
changed the title
|
This PR will be superceded by work on a related PR that adds a new form of priority calculation. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
Minimize inspection of large FASTA files by subsampling only with metadata and a sequence index and outputting only strain names for each subsample rule. Uses the new metadata-only filter interface proposed for Augur to collect the resulting subsampled strains and output a single FASTA file of all distinct strains.
As noted in 3216c5f, some subsample jobs require priority scores for another subsampled set and these score calculations require subsampled sequences. That same commit as a rule to extract only the subsampled sequences for a specific set as needed.
Although the goal of metadata-only filtering is to dramatically speed up this workflow, this PR does not address a major bottleneck in the current workflow which is the amount of time required by augur filter to write sequences to disk.
Related issue(s)
Related to nextstrain/augur#679
Testing
I've tested this workflow with the Nextstrain Europe build and also the multiple inputs example data.
CI tests will fail because this PR requires an Augur development branch (code that is not available in the Docker image). To test this PR locally, run Snakemake with the conda mode like so: