Tracking issue for function attribute #[coverage]

[richkadel]

This issue will track the approval and stabilization of the attribute coverage, needed to give developers a way to "hide" a function from the coverage instrumentation enabled by rustc -Z instrument-coverage.

The Eq trait in the std library implements a marker function that is not meant to be executed, but results in an uncovered region in all rust programs that derive Eq. This attribute will allow the compiler to skip that function, and remove the uncovered regions.

Unresolved questions

• is it ok that the attribute applied to the crate root, a function, or a module applies to all items within the marked item? So far we do not have such attributes other than the lint level attributes.

[richkadel]

cc: @tmandry @wesleywiser

[nagisa]

Nit: should probably follow the pattern of #[coverage(...)] much like #[inline] and many other attributes.

[Swatinem]

Just a wish that I have:
Instead of making this a function level attr, it should work equally on mod or even just blocks.

Another thing I wish for would be a crate-level attr that would disable coverage for every #[test] fn.

[richkadel]

Nit: should probably follow the pattern of #[coverage(...)] much like #[inline] and many other attributes.

I considered this, and discussed it in the initial PR. I'll copy those comments to this tracking issue, to make it easer to read and reply:

@tmandry said:

I also like the idea of combining the first two comments on #84605 by making it #[coverage(always/never)] and making some things (like tests?) disabled by default, but #[coverage(always)] could be used to override this. The only drawbacks I see are that #[coverage] on its own seems meaningless (unlike #[inline]) and these are longer to type.

The particulars of which things to make covered by default are a little hazy to me at this point, though. Really the problem I want to fix is that assert messages in tests appear uncovered, but that's kind of the point :)

@richkadel replied:

I also like the idea of combining the first two comments on #84605 by making it #[coverage(always/never)] and making some things (like tests?) disabled by default

Yes, I saw the comments, and I do want to consider them. Since this is feature-gated (and fixes a bug), I'd like to land this with no_coverage initially, and follow up with a different PR if we decide to change how it works.

I did consider something like #[coverage(never)], but (as you point out) I currently feel that #[coverage] or #[coverage(always)] doesn't really make sense to me.

In fact, I think coverage is especially relevant to tests, so I wouldn't want to disable it for tests by default, or require the user add the attribute in order to get test coverage.

The only other coverage-related attribute we may want to consider, in the future, is an attribute to enable coverage of unwind paths (#78544). But the MIR InstrumentCoverage pass isn't close to ready to support that yet.

It's definitely not uncommon for attributes to start with no_, and I just felt that no_coverage is much easier to understand and easier to remember compared to coverage(never) or coverage(none) or coverage(disable). (Also, never might be consistent with the terminology for inline, but the semantics are very different. inline has the notion of "best effort" or "when appropriate" inlining; but those semantics don't apply as well to coverage I think.)

@tmandry replied:

Agreed that coverage is relevant to tests, but I don't know if it's very relevant to the test functions themselves. But like I said, there might be a different way of getting at the problem I'm talking about.

In fact, your reply made me think of an approach I like better, which is removing counters that are along a path that always panics. We don't instrument the full panic paths anyway, and this is a straightforward analysis to do.

None of this is very relevant to this PR though! I just wanted to put my ideas down somewhere..

[richkadel]

Just a wish that I have:
Instead of making this a function level attr, it should work equally on mod or even just blocks.

Another thing I wish for would be a crate-level attr that would disable coverage for every #[test] fn.

@Swatinem - Can you expand on this? What are the use cases for disabling coverage at the crate level, or in crate tests?

For example, the use case addressed in PR #83601 is clear: The implementation of Eq required adding a special marker function (which gets added in every struct that derives from Eq) that was not intended to be called. Every struct that derived from Eq could never achieve 100% coverage. This unexecutable function clearly justifies the requirement for #[no-coverage].

I personally see the Eq trait issue as an exceptional case.

Generally speaking...

I think we want to keep the bar high for now (by restricting the scope of the attribute to functions, until any broader requirement is vetted by the Rust community).

So if there are other strong cases for excluding coverage instrumentation, please post them here so we can validate the use cases before expanding the scope of the coverage attribute.

[richkadel]

One more comment for clarity: PR #83601 has been merged and adds the feature-gated #[no_coverage] attribute. The attribute can be enabled on an individual function (by also adding #[feature(no_coverage]) or at the crate level.

[richkadel]

I should have added: This tracking issue will remain open until the attribute is stablized. This means there is still an opportunity to change the attribute, based on the tracking issue feedback.

Thanks!

[Swatinem]

@Swatinem - Can you expand on this? What are the use cases for disabling coverage at the crate level, or in crate tests?

This might be a bit personal preference, but is also done that way in other language ecosystems.
You generally want to have coverage for the code under test, not the test code itself. Having coverage metrics for test code provides zero value to me as developer.

[nagisa]

Note: there are previous initiatives to introduce inheritable/scoped/propagated attributes, e.g. for the optimize attribute, but implementing the propagation behaviour there seemed to be pretty involved to me at the time and IMO any attempts to introduce such attribute propagation mechanisms should be a standalone RFC introducing a mechanism in a more general way.

[jhpratt]

Would it be possible to add this attribute to the code generated by unreachable!()? Behind a flag, of course. Both #![feature(no_coverage)] and LLVM-based code coverage are unstable, so I don't think this would cause any issues unless I'm missing something.

[richkadel]

Unfortunately, #[no_coverage] (in its current form) applies to a function definition only. Applying that attributed before a function definition will cause the entire function to be ignored by coverage. But I assume you want the "call" to unreachable!() to be ignored by coverage. Ignoring a statement or block is not currently supported.

I can understand the motivation for this request, and you may want to file a new issue, if it doesn't already exist. There is a potential downside: coverage reports would not be able to show that an unreachable block was unintentionally executed. Maybe that's a minor tradeoff, but the current coverage reports confirm that unreachable blocks are not reached, by showing the unreachable block has a coverage count of zero.

I think adding support for #[no_coverage] blocks may be the best solution, but macro expansion may make this tricky to get right. There may not be a quick solution.

[richkadel]

And I may be wrong about block #[no_coverage] being the "best solution". Thinking about this just a bit more, adding the #[no_coverage] attribute at the function level was not that difficult, because AST-level functions map directly to MIR bodies (and coverage is instrumented within each MIR body). That's easy to turn on or off at the function level. But AST-level blocks don't translate directly to internal MIR structure. That will also make implementing more granular support difficult, in addition to macro expansion complications.

[tmandry]

To me, #[no_coverage] blocks seem like the best solution from a language perspective. Agreed that there might be some work threading things through on the implementation side.

For macros, we can just think about this as applying to a particular expansion (and the attribute applies to the block in HIR, i.e. after macro expansion). Maybe there are other issues I'm not thinking about.

[jhpratt]

I don't think accidentally executing an "unreachable" expression is an issue for coverage, as that's not the job of coverage; tests are where that should be caught.

There is a tracking issue for this already, I wasn't sure if there was an easy-ish way to land it. Apparently that's not the case. I would imagine in the future statement and expression-level masking for coverage purposes will exist, which would necessarily allow for this. Ultimately this would just cause the macro to include the attribute in its expansion.

[clarfonthey]

Is there a good reason why derived traits are included in coverage reports, and not also marked as #[no_coverage]? Considering how the assumption for derived traits is that they're already "tested" by the standard library, I figure that it makes the most sense to simply not include coverage for them by default.

But, I could see an argument for including them by default but allowing a #[no_coverage] attribute on struct definitions to disable them, if you feel like going that route.

[clarfonthey]

In terms of specifically how this attribute should be named, I would be in favour of going with a #[coverage(...)] style attribute as @nagisa recommended. This also opens up adding other coverage-based attributes in the future, like:

• #[coverage(hide)] -- the existing #[no_coverage] attribute
• #[coverage(ignore)] -- would mark code as "coverage doesn't matter," e.g. derives, telling tools to still show coverage for lines but not including it in total coverage metrics
• #[coverage(disallow)] -- would mark code as "should not be covered", e.g. the unreachable!() macro, allowing tools to show warnings when they are covered
• #[coverage(require)] -- would mark code as "should always be covered", allowing tools to show warnings when they are not covered

This obviously wouldn't be required for an initial stable #[coverage] attribute, but I think in particular the distinction between #[coverage(hide)] and #[coverage(ignore)] might be useful to have. Something like the Eq method should be marked #[coverage(hide)] because it's more confusing to show coverage than not, but in general I think it would make sense to mark other stuff as #[coverage(ignore)] in general.

[richkadel]

I'm not convinced ignoring trait functions is a good idea, from a test coverage perspective.

When you derive a trait, your struct's API contract has changed. That struct now supports every function defined in that trait. To guarantee 100% test coverage of the behaviors of every function for your struct, you would have to test it.

Without adding a test, the "ignore" concept is really just a cheat, because you want to achieve 100% test coverage without actually testing it 100%. You can say you are "really confident" that it will work fine, but you can't prove it without some kind of test analysis or observed behavior, via a test.

[clarfonthey]

Yes, the API contract changes when you derive standard traits, but I personally don't believe that testing them adds any value for the developer. For example, assert_eq!(value, value.clone()); would provide coverage for derived PartialEq and Clone implementations, but as far as your library goes, you're not really testing anything important; the only way those would fail is if there's an error in the compiler, which I don't think is everyone's responsibility to test.

I should clarify I'm explicitly talking about the standard derives; individual proc macros can make the decision whether they want to add the attribute to their generated code or not.

[clarfonthey]

That's fair; I'll remove the guarantees about instrumentation being added since, although I'd like to have some sort of basic guarantee like "if the functions are run in the final executable, they will have coverage instrumentation" but it feels like the amount of caveats you have to add to make that work is not worth it at the moment.

[bossmc]

Are there known cases where the instrumentor won't create a counter for a function marked for coverage? If the coverage counter as created and added to the coverage map it doesn't matter if dead-code elimination or link-time GC removes the function, the coverage will still correctly report that the function wasn't hit.

[clarfonthey]

From what I understand, coverage is instrumented after dead code elimination, so, dead code won't ever be instrumented.

What you're saying makes sense as a solution, but I'm not sure it's possible with LLVM or the current setup, and it's better to just explain the state of coverage now than block the attribute on it being improved.

[bossmc]

I wasn't suggesting to hold up the implementation, I was just surprised there were cases where instrumentation (or at least, counters) weren't emitted for a function when coverage was enabled.

Though I'm surprised by your assertion, as I understand it, coverage markers are added by the frontend, whereas DCE is surely done by the backend (along with coverage-lowering which is uninteresting for this question)?

[clarfonthey]

Yeah, it's a very confusing system for sure.

My understanding is that the entire thing is done after code generation, meaning that effectively, only the code that ends up in the final executable will show up on the list of coverage markers. The compiler can add in extra annotations to source that tell the code generator how coverage should be set up, but ultimately, those markers are ignored until the final code is generated and the final list is made.

This is why so many suggestions about coverage involve disabling LTO, disabling dead code removal, etc. Because ultimately, the original source doesn't matter for coverage, and the instrumentation just tracks back to the original code from the code that's left at the end.

[bossmc]

That's true for things like gcov and kcov, where coverage data is reverse engineered from the compiled binary, but the whole point of frontend-driven coverage is that the compiler knows how to refer to the source code to create coverage data (see https://clang.llvm.org/docs/SourceBasedCodeCoverage.html and https://rustc-dev-guide.rust-lang.org/llvm-coverage-instrumentation.html which both make it clear that producing the coverage map and injecting the llvm.instrprof.* intrinsics into the IR - those intrinsics are then lowered by LLVM to update the counters, but the counters default to 0 so if that code gets stripped the counter will report that the code wasn't hit).

In fact a quick prototype shows that dead code is still reported as missed so maybe this is working today anyway?

$ rustc +nightly -C instrument-coverage src/main.rs -O
$ ./main
$ llvm-profdata-18 merge -sparse default_*.profraw -o default.profdata
$ llvm-cov show ./main --instr-profile default.profdata
    1|      1|fn main() {
    2|      1|    println!("Hello, world!");
    3|      1|}
    4|       |
    5|      0|fn wibble() {
    6|      0|    println!("Wibble!");
    7|      0|}

Edit: and yes, the wibble function was indeed stripped from the binary:

$ readelf --symbols --wide ./main | rustfilt | grep wibble
$ strings ./main | grep Wibble
$

[Zalathar]

In my mind, the main thing we are stabilizing is the syntax of #[coverage(off)] and #[coverage(on)], so that they can be used in stable code without a compiler error.

The precise effect of those attributes should be left to the discretion of the compiler implementation, in much the same way that -Cinstrument-coverage itself stably exists, while its actual details are not stable.

(We can still document the intended meaning of those attributes, and in practice the compiler will make a good-faith effort to honour that as much as reasonably possible. But we shouldn't be making any stable promises about which functions do or don't get instrumented, just as we don't make any stable promises about how the code inside those functions is instrumented.)

[Zalathar]

In the current implementation, there are three main reasons why a function might not show up in the final coverage output:

• The compiler decides that it can't/shouldn't be instrumented (e.g. #[coverage(off)], #[naked], certain functions whose bodies are clearly unreachable).
• The compiler tries to instrument a function, but can't find any spans worth instrumenting, so it gives up.
  • It's hard for this to happen in “normal” code, but macros make things a lot more complicated.
• The compiler does instrument a function, but somehow that function is omitted by codegen (e.g. due to a bug in the handling of coverage metadata for unused functions).

[Zalathar]

An important note for T-lang raised by Oli on the implementation PR:

so the intent is for nested modules also to use the attribute from the outer module? This is a new concept in Rust iirc, so maybe add it as an important point on the tracking issue for T-lang to talk about during stabilization

Also copying my own response from there:

The current behaviour (scanning all the way up to the crate root) wasn't my original intention, but as I implemented the feature it ended up being (a) easier, and (b) plausibly the right design, so I decided to stick with it until/unless the team decides otherwise.

[Zalathar]

As far as I'm aware, the closest precedent for my proposed behaviour (scanning all enclosing functions/impls/modules up to the crate root) is the current behaviour of the lint-control attributes (e.g. #[allow(..)]).

So what's new here is extending that sort of behaviour to a non-diagnostic attribute.

(It's slightly surprising that this hasn't come up before, but looking through the existing built-in attributes, there isn't much demand for recursive behaviour outside of lints.)

[clarfonthey]

I'm also surprised this is the first place this has come up, but it also feels like precisely the right behaviour, so. 🤷🏻

[obi1kenobi]

I believe #[deprecated] and #[doc(hidden)] also have such behavior. Marking a module deprecated or hidden causes all its contents to be deprecated or hidden accordingly. Same for impl blocks etc. I'm pretty sure I added test cases in cargo-semver-checks to make sure this is the case and doesn't change in the future 😁

As an interested user of this attribute, this behavior makes sense to me 👍 Thank you to everyone helping make it happen!

[joshtriplett]

@rfcbot resolved should-attribute-disable-coverage-for-nested-functions?

[joshtriplett]

@rfcbot resolved should-attribute-disable-coverage-for-inlined-functions?

[nikomatsakis]

@rfcbot reviewed

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[traviscross]

@rustbot labels -I-lang-nominated

This is in FCP, so we can unnominate. We discussed this briefly in the extended triage meeting today.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[PoignardAzur]

This might come late in the discussion, but have you considered stabilizing coverage as a diagnostic namespace attribute? E.g.:

#[diagnostic::coverage(off)]
fn foobar() {}

The main advantage of doing that is that people could then use coverage annotations while keeping a MSRV of 1.78.0. Same thing for future changes to the feature, if any.

And I do think coverage as a diagnostic makes sense. Yes, it changes the emitted artifact, but it doesn't change the program semantics. And more importantly, it doesn't affect dependencies: if a dependency has coverage annotations you don't understand, you can just ignore them without negative impact to your output.

[clarfonthey]

Coverage isn't a diagnostic, though; it's a build flag that can change the way the crate is built under certain circumstances.