Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing: Prioritize changed unit tests #2832

Merged
merged 4 commits into from
Jan 19, 2023

Conversation

WardBrian
Copy link
Member

Summary

This adds a short round of testing which is run in parallel with the header checks. It sees what files are changed relative to develop and runs only the tests which either changed, or the tests of the functions which changed, if it can determine them.

The goal here is to make the math CI pipeline fail faster and with a more localized error for simple PRs. Obviously if you change a function which other functions depend on it might still fail in later stages, but this should help with iteration.

Note that these tests do get re-run later during the stage they normally would.

Other notes: OpenCL tests are excluded, and if more than 20 tests have changed we skip this to avoid it slowing down the whole CI pipeline - at that point, you probably want to move on to the later full stages anyway.

Thanks to @SteveBronder for the idea

Checklist

  • Math issue #(issue number)

  • Copyright holder: (fill in copyright holder information)

    The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
    - Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
    - Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

  • the basic tests are passing

    • unit tests pass (to run, use: ./runTests.py test/unit)
    • header checks pass, (make test-headers)
    • dependencies checks pass, (make test-math-dependencies)
    • docs build, (make doxygen)
    • code passes the built in C++ standards checks (make cpplint)
  • the code is written in idiomatic C++ and changes are documented in the doxygen

  • the new changes are tested

@WardBrian
Copy link
Member Author

Thanks to some debugging with @serban-nicusor-toptal this is now working. See: https://jenkins.flatironinstitute.org/blue/organizations/jenkins/Stan%2FMath/detail/PR-2832/17/pipeline/165/

It correctly ran both the modified test abs_test, as well as the prim AND mix tests for asinh

@WardBrian WardBrian force-pushed the tests/prioritize-changed-tests branch from 437e8ad to 6b65891 Compare October 24, 2022 15:17
@WardBrian
Copy link
Member Author

@rok-cesnovar This is ready for review now, got the merge conflicts fixed up after the other CI changes

Copy link
Member

@syclik syclik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Awesome. Thanks!

@WardBrian WardBrian merged commit 2f6f426 into develop Jan 19, 2023
@WardBrian WardBrian deleted the tests/prioritize-changed-tests branch January 19, 2023 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants