Skip to content

Commit

Permalink
ci: Faster coverage tests (#545)
Browse files Browse the repository at this point in the history
Copies the idea from `hugr`. 

For the rust checks, the CI ran nightly tests followed by the same tests
with coverage instrumentation. This added unnecessary delay to the
coverage report.
This PR merges the two steps, so the rust codecov analysis should appear
~2:30 minutes faster.

For python, running with and without coverage takes almost the same
time. The PR merges the test runs, but still leaves the codecov upload
in a separate job (outside the critical path of the required checks).
This saves ~1:30.

Before (Coverage delay ~5mins):
<img width="690" alt="image"
src="https://github.com/user-attachments/assets/2139a314-258e-4212-92b4-f291f5a52e4e">
After (Coverage delay ~2mins):
<img width="691" alt="image"
src="https://github.com/user-attachments/assets/92c28af2-c5ab-4a10-981a-9ad94c9e80da">
  • Loading branch information
aborgna-q authored Aug 16, 2024
1 parent d0d2629 commit 6e3dd57
Showing 1 changed file with 30 additions and 35 deletions.
65 changes: 30 additions & 35 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,9 +163,9 @@ jobs:
strategy:
fail-fast: true
matrix:
# Pinned nightly version until this gets resolved:
# https://github.com/rust-lang/rust/issues/125474
rust: ['1.75', beta, 'nightly-2024-05-22']
# Stable is covered by `tests-stable-no-features` and `tests-stable-all-features`
# Nightly is covered by `tests-nightly-coverage`
rust: ['1.75', beta]
name: tests (Rust ${{ matrix.rust }})
steps:
- uses: actions/checkout@v4
Expand All @@ -185,40 +185,18 @@ jobs:
- name: Tests with all features
run: cargo test --verbose --workspace --all-features

tests-py:
tests-nightly-coverage:
needs: changes
if: ${{ needs.changes.outputs.python == 'true' }}
runs-on: ubuntu-latest
name: tests (Python)
steps:
- uses: actions/checkout@v4
- uses: mozilla-actions/sccache-action@v0.0.5
- name: Install stable toolchain
uses: dtolnay/rust-toolchain@stable
- name: Install poetry
run: pipx install poetry
- uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'poetry'
- name: Build pyo3 bindings
run: |
poetry install
poetry run maturin develop
- name: Test pyo3 bindings
run: poetry run pytest

coverage-rs:
name: Check Rust coverage 🦀
needs: [changes, tests-rs-stable-no-features, tests-rs-stable-all-features, tests-rs-other, check-rs]
# Run only if there are changes in the relevant files and the check job passed or was skipped
if: always() && !failure() && !cancelled() && needs.changes.outputs.rust == 'true' && github.event_name != 'merge_group'
# Run only if there are changes in the relevant files
if: ${{ needs.changes.outputs.rust == 'true' && github.event_name != 'merge_group' }}
runs-on: ubuntu-latest
name: tests (Rust nightly, coverage)
steps:
- uses: actions/checkout@v4
- uses: mozilla-actions/sccache-action@v0.0.5
- uses: dtolnay/rust-toolchain@master
with:
# Nightly is required to count doctests coverage
toolchain: 'nightly'
components: llvm-tools-preview
- name: Install cargo-llvm-cov
Expand All @@ -238,12 +216,11 @@ jobs:
flags: rust
token: ${{ secrets.CODECOV_TOKEN }}

coverage-py:
name: Check Python coverage 🐍
needs: [changes, tests-py, check-py]
# Run only if there are changes in the relevant files and the check job passed or was skipped
if: always() && !failure() && !cancelled() && needs.changes.outputs.python == 'true' && github.event_name != 'merge_group'
tests-py:
needs: changes
if: ${{ needs.changes.outputs.python == 'true' }}
runs-on: ubuntu-latest
name: tests (Python)
steps:
- uses: actions/checkout@v4
- uses: mozilla-actions/sccache-action@v0.0.5
Expand All @@ -261,6 +238,24 @@ jobs:
poetry run maturin develop
- name: Run python tests with coverage instrumentation
run: poetry run pytest --cov=./ --cov-report=xml
- name: Upload coverage output artifact
uses: actions/upload-artifact@v3
with:
name: py-coverage
path: coverage.xml

coverage-py:
name: Upload Python coverage 🐍
needs: [changes, tests-py, check-py]
# Run only if there are changes in the relevant files and the check job passed or was skipped
if: always() && !failure() && !cancelled() && needs.changes.outputs.python == 'true' && github.event_name != 'merge_group'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v3
with:
name: py-coverage
path: coverage.xml
- name: Upload python coverage to codecov.io
uses: codecov/codecov-action@v4
with:
Expand Down

0 comments on commit 6e3dd57

Please sign in to comment.