Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP, ENH: add docstring cov to html #808

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tylerjereddy
Copy link

One of the items on the wishlist for testing docstrings in NumPy / SciPy is the ability to report "coverage" for those docstrings that are probed by doctesting infrastructure. In particular, I had in mind the idea that labels in the html coverage reports that one often finds as part of the CI for large OSS scientific projects might be expanded to include hits on docstrings.

This lead to me to a simple example on StackOverflow by @rhettinger, which really got me excited even more. However, I noticed that executing the exact same code and trying to display with coverage html effectively ignores the docstring using both stable and master branch of coveragepy. I dug a little deeper and noticed that coverage annotate actually has a much shorter / simpler loop for doing the labelling vs. html, and I figured I'd take a look at the html assignment of lines to label classes, and see if I could get it to do what I want.

This PR has a draft of what I had in mind that I initially thought was working, but actually seems to be not quite right yet. There are probably various ecosystem considerations & things like trying to fuse these results with results that don't have docstrings labelled, etc. But if I I can't get this "simple" case to work, the prospects for expanding the "feature" usage to large packages wouldn't be great.

With this feature branch and my .coveragerc initially looking like this:

[html]
docstring_display = True

with the original example code from StackOverflow we get the docstring now highlighted along with the source itself that was ultimately executed by doctest:

image

and with a doctest SKIP directive added in to Raymond's example, another design question comes up--would it be possible / worth the trouble to have a separate highlight color / category for docstring examples that are not executed? (notice that the source proper is still correctly marked as not executed, since the docstring didn't get executed, although the docstring was "considered but skipped" so perhaps technically "covered"):

image

So, as I was writing this up, I thought "hey this is working just like the annotate case so must be good," but then I commented out the doctest execution and it seems the docstring lines are just getting labelled as a fallback, not because the tracing infrastructure knows that they are getting executed:

image

Ok, so maybe I got excited a little too early. That got me curious about Raymond's original example with annotate. If I turn off the docstring execution there by commenting out the doctest.testmod() line I get a result suggesting that docstring hits are not actually meaningful results from coveragepy tracing / interpretation infrastructure:

> def linear(a, b):
>     ''' Solve ax + b = 0
  
>         >>> linear(3, 5)
>         -1.6666666666666667
  
>     '''
!     if a == 0 and b != 0:
!         raise ValueError('No solutions')
!     return -b  / a

This suggests to me that there may be another "meta" level to the tracing if we want to handle doctest probing of those string lines as "hits." That might be split into two categories:

  1. was the string "probed" by doctest infrastructure?
  2. was a line in that string exec(compile()) run by doctest or subclassed doctest infrastructure in a library like NumPy / SciPy?

* provide a config file option that
allows docstrings probed by doctest
to be highlighted in html reports
@nedbat
Copy link
Owner

nedbat commented Aug 4, 2019

Sorry I've left this for so long: as I understand it, the goal is to mark docstrings as executed if doctest executes them? I'm not sure what hooks doctest gives us for knowing which are executed. Or do the line numbers get reported as executed when doctest runs them? I'll have to do some experiments.

In the examples you show, I'm not sure why line 2 is marked as executed? It's not executable code, which makes me think the entire docstring is marked executed even if only part of it is.

@tylerjereddy
Copy link
Author

Sorry I've left this for so long: as I understand it, the goal is to mark docstrings as executed if doctest executes them?

Yes, that would be cool, but as you note it is not immediately clear how to trace through the "meta-execution" that doctest in CPython does. I think I saw an explicit exec when I took a peek inside doctest, but I don't know if that's enough to trace through.

@ProsperousHeart
Copy link
Contributor

Is this still a valid PR? I know it's in draft format, but it's a few years old at this point with several updates since then

@tylerjereddy
Copy link
Author

I think the idea is good and still not supported, but this PR is probably useless beyond requesting the feature at this point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants