filter: Reorganize Cram test files

[victorlin]

Context

The current filter.t is pretty cumbersome to work with. From a Slack thread:

Say you want to add a new test at the end of the file. It's really inefficient to wait ~1m for all other tests in the file to see the outcome of your new test, which can run in milliseconds. Instead, you want to run just the new test. For pytest this isn't so bad – there are easy ways to avoid running parts of the file (e.g. comment/uncomment existing lines). However, for cram, I find myself: removing all the lines up to the top where some setup stuff happens, working on a test, then git restore-ing the removed lines. It's not terrible, but a little more friction than simply commenting/uncommenting.

Description of proposed changes

This change breaks that file into several smaller files.

• Split filter.t into much smaller files:
  • For most files, each is setup + one augur filter command.
  • A few files also check the output or run related commands.
• Re-organize supporting files so that everything is under tests/functional/filter/, which has two folders:
  • cram: individual test files
  • data: supporting data (previously directly under tests/functional/filter/)

Some noticeable differences:

• Easy to have a high-level overview of tests:

  % ls -1 tests/functional/filter/cram/
  _setup.sh
  filter-exclude-include.t
  filter-metadata-duplicates-error.t
  filter-metadata-not-found-error.t
  filter-metadata-sequence-strains-mismatch.t
  ...
  

• Progress dots and numbers on test results are more meaningful:

  % cram --shell=/bin/bash tests/functional/filter/cram/
  ........................
  # Ran 24 tests, 0 skipped, 0 failed.
  

• Easy to run just one test:

  % cram --shell=/bin/bash tests/functional/filter/cram/filter-no-outputs-error.t 
  .
  # Ran 1 tests, 0 skipped, 0 failed.
  

• There is more setup overhead for each test, but in terms of run time it's only a 1 second difference:

  % time cram --shell=/bin/bash tests/functional/filter.t    
  .
  # Ran 1 tests, 0 skipped, 0 failed.
  cram --shell=/bin/bash tests/functional/filter.t  30.09s user 5.89s system 135% cpu 26.567 total
  

  vs.

  % time cram --shell=/bin/bash tests/functional/filter/cram/
  ........................
  # Ran 24 tests, 0 skipped, 0 failed.
  cram --shell=/bin/bash tests/functional/filter/cram/  30.28s user 6.16s system 132% cpu 27.598 total
  

Related issue(s)

N/A

Testing

See CI.

[huddlej]

Cool! I like the new layout and test filenames. We'll need to update both the CI test runner and the run_test.sh script to search for Cram tests in a smarter way. Actually, I just noticed (thanks to your example code) that Cram has its own smart search built in so this works:

cram --shell=/bin/bash tests/

[victorlin]

@huddlej good point, done: 8deac07

[victorlin]

woo I'm so excited to use this! No more merge conflicts from adding to filter.t in multiple PRs.

If this works well, I'll do the same for other large .t files.

[tsibley]

Seeing this late, but woo, this is a great direction!