[wip]: Add the functionality of join metadata and clades

[j23414]

Description of proposed changes

After some discussion with @joverlee521, moved join-metadata-and-clades.py from PR: #20 to this draft.

Some of the functionality may be replaced by csvtk but there are customized calculations in certain pathogen repositories.

mpx/join-metadata-and-clades.py
        rsv/join-metadata-and-clades.py       3 lines different
        dengue/join-metadata-and-clades.py      IDENTICAL
        ncov/join-metadata-and-clades     114 lines different

This is a placeholder that the functionality of join-metadata-and-clades requires more discussion and thought.

Related issue(s)

Subset of scripts listed in #1

Checklist

• Checks pass
• If adding a script, add an entry for it in the README.

[j23414]

We could combine the two files without performing complex calculations by using a combination of csvtk rename and tsv-join as follows:

# Rename columns in the input.nextclade file
cat {input.nextclade} \
| csvtk -tl rename2 \
  -F \
  -f '*' \
  -p '(.+)' \
  -r '{{kv}}' \
  -k {input.nextclade_field_map} \
  > results/nextclade_renamed.tsv
  
# Join the renamed nextclade file with the input.metadata file
cat {input.metadata} \
 | tsv-join -H \
 --filter-file results/nextclade_renamed.tsv \
 --key-fields seqName \
 --data-fields accession \
 --append-fields `awk '{print $2}' results/nextclade_renamed.tsv | tr '\n' ','` \
 --allow-duplicate-keys \
 --write-all -1 \
 > {output.metadata}

[joverlee521]

@j23414 Would you be up for replacing join-metadata-and-clades in monkeypox with your csvtk/tsv-join example? It would be nice to do a full test run there to see how the outputs compare.

[huddlej]

I found myself needing to implement something similar in the flu_frequencies workflow and appreciated csvtk join's option to specify different names for the join columns in the two inputs (e.g., seqName in Nextclade input and strain in metadata). I originally planned to use tsv-join, but I was frustrated by the need to rename columns to have a matching key column name in both inputs.

Independent from the tools you end up using here, you can drop the cat block from the code above such that this:

cat {input.nextclade} \
| csvtk -tl rename2 \
  -F \
  -f '*' \
  -p '(.+)' \
  -r '{{kv}}' \
  -k {input.nextclade_field_map} \
  > results/nextclade_renamed.tsv

becomes this:

csvtk -tl rename2 \
  -F \
  -f '*' \
  -p '(.+)' \
  -r '{{kv}}' \
  -k {input.nextclade_field_map} \
  {input.nextclade} > results/nextclade_renamed.tsv

[j23414]

Closed since this script is replaced with csvtk and tsv utils.

• nextstrain/mpox@d053086