read_metadata delimiter detection is fragile

[huddlej]

Current Behavior

read_metadata tries to load a given file as tab-delimited data only if the file extension is .tsv. Any other extensions get parsed as comma-separated values.

Expected behavior

read_metadata should provide a more user-friendly interface by more intelligently detecting the metadata delimiter. Users should be able to provide a tab-delimited metadata file like metadata.txt and have this data loaded properly. Similarly, the .tab extension is a common one for tab-delimited data, but files with this extension will get parsed as CSV files now.

How to reproduce

Steps to reproduce the current behavior:

1. Create a tab-delimited metadata file named metadata.txt
2. Run any augur command that loads metadata with this file (e.g., augur filter --sequences sequences.fasta --metadata metadata.txt --min-date 2010-01-01 --output filtered.fasta

Possible solution

Consider using pandas's builtin delimiter detection by setting sep=None and engine="python" when reading in metadata. Remove extension-based attempts to infer delimiters.

Using the python engine to load data is slightly slower than the C engine, but the user interface improvement is worth the performance cost.

Additional context

This issue was originally raised on the Nextstrain discussion board.

[swarris]

I took me about a day and an inspection of the source code to find out why it would not load my tab delimited file.... Apparently because the code assumed a comma, not a tab... Without informing me. Also the documentation has no mentioned about this. Please remove this separator switch, or at least inform the user about this.

[huddlej]

We're sorry about the headaches caused by the original delimiter detection, @swarris! This issue should now be resolved in the master branch and will be part of our next release (v 10.0.0). If you find that these changes don't fix the problem for your data, you can reopen this issue and we'll follow up with you.