Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add details about testing and polish dev doc #481

Merged
merged 1 commit into from
Mar 26, 2020

Conversation

willingc
Copy link
Contributor

This PR adds details to the testing section of the developer docs as well as general
edits to improve understanding for new contributors. Specifically, the changes
include:

  • Add document contents at the top of the file for reader convenience
  • Add headers as needed to reflect the contents
  • Edit some prose to improve contributor understanding
  • Make the markdown style more consistent for code blocks and spacing

Closes #480

@jameshadfield
Copy link
Member

jameshadfield commented Mar 26, 2020

Thanks @willingc 👍 this all looks good to me. Two questions:

  1. From my point of view, encouraging tests which cover the entire augur subcommand would be valuable. As an example, for augur filter there could be tests which run augur filter --sequences <s> --max-length <x> where <s> is a small test file and then check that the sequences written out by the command are the ones we expect. Is this common practice? Can pytest help us with this? It seems to me that this would be valuable before we refactor the internals too much.

  2. The "e2e" tests (not sure of precise terminology here) mentioned in Add testing guide #480 aren't included in this PR. These have been critical for me developing augur & auspice to ensure augur is generally working as expected (and that augur commands don't exit unexpectedly). Is there a way to improve this approach which we could recommend here?

(If these questions are controversial I'm happy for them to be made into separate issues and merge this PR as-is.)

@tsibley
Copy link
Member

tsibley commented Mar 26, 2020

Re: testing the CLI interface itself, I agree that's very important. pytest cases can definitely be written to do this. There are also targeted testing tools for CLIs which could be integrated into the test suite. I've used cram in the past and really liked it.

@huddlej
Copy link
Contributor

huddlej commented Mar 26, 2020

This looks great, @willingc! Thank you!

It looks like folks are using pytest-mock to test the command line interface, too. As we get more of these tests merged, those should serve as documentation for how folks should write future tests.

I'm going to merge this, but if we can add explicit requests for CLI testing if these types of tests don't end up happening.

@huddlej huddlej merged commit b17e78b into nextstrain:master Mar 26, 2020
@tsibley
Copy link
Member

tsibley commented Mar 27, 2020

@huddlej FYI, the pytest-mock example you linked is being used to mock the internal run_shell_command() function in Augur, so that it doesn't actually do anything. The tests just check that it was called with expected arguments. This doesn't test the Augur CLI itself.

@willingc
Copy link
Contributor Author

@tsibley @huddlej Absolutely the CLI should be covered. My recommendation would be to set up a code coverage tool similar to codecov.io. This would allow us to display code coverage on every PR. It's helpful for those submitting PRs too.

As for testing, I would recommend getting the coverage up with unit tests as quickly as possible and then have contributors begin writing integration tests which will have more end to end testing.

@willingc willingc deleted the iss-480 branch March 27, 2020 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add testing guide
4 participants