-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make tests run faster #2266
Make tests run faster #2266
Conversation
LOL, I was also working on something similar 😅. I'm gonna have a look!!! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I totally agree that test time MUST be small. Our test suite was starting to last too long and this must be fixed.
On the other hand, I think we should increase our test coverage: I think there are many parts of the code without being tested and this must be also addressed.
I would suggest dividing our test suite into real unit tests (super fast) and integration tests (slower tests):
- for real unit tests, we can be exhaustive and try to increase the coverage, as they are really fast
- for integration tests, we must be more cautious and we should select some general cases and only test those those combinations
- we could even keep some end-to-end tests (like
load_dataset
), but very few
I think this PR goes in the right direction, but we should continue improving real unit test (at high speed) coverage.
Feel free to tell me your opinion and how we could keep delving into this matter.
Sorry I didn't know you were also working on it ^^' |
Never mind: we both noticed tests can be improved. More PRs to come... 😉 According to the literature, unit tests are those that test a behavior unit, isolated from the other components and must be very fast: for me, this last requirement implies that they must be performed completely in memory. As opposed, integration tests are those which also test interactions with external components, like web services, databases, file system, etc. The problem I see is that our code is still too coupled and it is difficult to isolate components for testing. Therefore, I would suggest acting iteratively, by refactoring to decouple components and then implement unit tests for each component in isolation. |
From 7min to 2min to run pytest.
Ideally we should keep the whole CI run time below 10min.
In this PR I removed the remote tests that were never used.
I also replaced nested parametrized tests with unit tests.
This makes me think that we could still add more high level tests to check for a few combinations of parameters (but not all of them since there are too many of them).
Let me know what you think
Finally in another PR we can also separate in two circleci jobs: