io: Parse metadata with C engine, restrict to either CSV or TSV

[victorlin]

Description of proposed changes

See commit messages.

Related issue(s)

Thinking about this since I'm making similar changes for the augur filter database implementation.

Testing

• Test added
• Checks pass

Checklist

• Add a message in CHANGES.md summarizing the changes in this PR that are end user focused. Keep headers and formatting consistent with the rest of the file.

[fanninpm]

You could also use csv.Sniffer.sniff() to return a Dialect, which can be passed in to pandas.read_csv() as the dialect keyword argument.

[tsibley]

The only concern would be if anyone is leveraging the Python parser for a non-tab-delimited file (e.g. metadata.csv). This would be a breaking change.

I'm not sure if this is an acceptable trade off. Diving into history, Augur's supported CSV for metadata since at least 2018 and delimiter sniffing since mid-2020.

@huddlej may have thoughts here too.

[huddlej]

I agree, @tsibley, that we can't drop CSV support. The original context for the current implementation is described in #574. There are two separate problems:

1. determining the delimiter of an input file
2. efficiently parsing a file given its delimiter

In the older Augur implementations, we addressed problem 1 by inspecting the extension of the input filename. This led to the problems in #574. We opted for the convenience of pandas's delimiter sniffer in Python parser mode, to solve this problem at the expense of a slower solution to problem 2.

As @fanninpm points out, we could use csv.Sniffer directly on the first line of the input file in read_metadata. Once we know the delimiter, we could call read_csv with that delimiter and use the C engine to parse. This would solve both problems without breaking backward compatibility.

[victorlin]

Thanks @fanninpm @tsibley @huddlej for the comments and suggestions. I finally updated this PR to use csv.Sniffer.

[codecov]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.02 🎉

Comparison is base (1dacac1) 68.39% compared to head (9f48ff2) 68.42%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #812      +/-   ##
==========================================
+ Coverage   68.39%   68.42%   +0.02%     
==========================================
  Files          63       63              
  Lines        6812     6818       +6     
  Branches     1671     1672       +1     
==========================================
+ Hits         4659     4665       +6     
  Misses       1843     1843              
  Partials      310      310              

Impacted Files	Coverage Δ
augur/io/metadata.py	96.17% <100.00%> (+0.15%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[joverlee521]

LGTM!

[victorlin]

@joverlee521 I pushed a couple touch-ups (70d8150...740a330), do those LGTY?

[joverlee521]

@victorlin Changes look good! (Ignoring the unrelated failing Cram test)