Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish C-level coverage of the CPython test suite #94759

Open
mdboom opened this issue Jul 11, 2022 · 0 comments
Open

Publish C-level coverage of the CPython test suite #94759

mdboom opened this issue Jul 11, 2022 · 0 comments
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@mdboom
Copy link
Contributor

mdboom commented Jul 11, 2022

Feature or enhancement

Automatically publish C-level coverage of the CPython test suite.

Pitch

Recent issues, such as the problems with frame.setlineno, have revealed that the C-level coverage of the test suite isn't perfect. While source-level coverage isn't the only metric of a good test suite, paying closer attention to it should help us have more confidence when making changes to the CPython core.

Proposed design for MVP

  • Publish results nightly: This cadence should be enough for most cases. While various tools like codecov.io can run on every pull request, coverage takes a long time to run, and IME, there is enough noise in the results that developers (especially new to the project) would spend a lot of time "chasing" coverage.
  • Use clang/llvm-cov rather than gcc/gcov: As @erlend-aasland pointed out the LLVM toolchain coverage provides branch counts and coverage of subexpressions which seems pretty useful. There is a risk that this isn't the same tooling as most Linux developers would use, and not what make coverage-report does.
  • Publish to Github Pages: The easiest thing seems to be to publish to Github Pages in a new repository in the python org for this purpose. This keeps all of the credential management within Github.

Out of scope

Creating diffs between coverage runs seems like it would be useful, but I haven't found a good tool for that yet.

Python-level coverage reports would likewise be useful, but should be a follow-on to this.

Previous discussion

@brettcannon had set up coverage reports in the past, but they have been broken for a couple of releases. This aims to be simpler than that approach by using a pure Github stack (not using codecov.io) and would just run nightly, not as part of every pull request.

More recent discussion began here: faster-cpython/ideas#426

@mdboom mdboom added the type-feature A feature request or enhancement label Jul 11, 2022
@mdboom mdboom changed the title Public C-level coverage of the CPython test suite Publish C-level coverage of the CPython test suite Jul 11, 2022
@AlexWaygood AlexWaygood added the tests Tests in the Lib/test dir label Jul 11, 2022
tiran added a commit to tiran/cpython that referenced this issue Jul 12, 2022
- fixed issue linker flags, LIBS is not passed to BLDSHARED
- include version and git commit in output
- include "Includes" directory, headers now have inline functions
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 13, 2022
(cherry picked from commit f5c02af)

Co-authored-by: Christian Heimes <christian@python.org>
miss-islington added a commit that referenced this issue Jul 13, 2022
(cherry picked from commit f5c02af)

Co-authored-by: Christian Heimes <christian@python.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
None yet
Development

No branches or pull requests

2 participants