Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fine grain output verbosity #11387

Open
plannigan opened this issue Sep 4, 2023 · 5 comments
Open

Fine grain output verbosity #11387

plannigan opened this issue Sep 4, 2023 · 5 comments
Labels
topic: reporting related to terminal output and user-facing messages and errors type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature

Comments

@plannigan
Copy link
Contributor

What's the problem this feature will solve?

There are multiple open issues that are requesting more control over the output displayed by pytest. --verbose turns everything up, but it would be nice to be able to just increase or decrease the verbosity for specific aspects of the output.

Open issues that I've found that fit into this problem space (there may be others):

Describe the solution you'd like

I think a way that could resolve all of these is to add an abstraction layer on top config.option.verbose. Rough pseudo-code example:

class OutputVerbosity:
  def __init__(self, config_options):
     self._options = config_options
     self.verbose = self._options.verbose

  def verbosity_for(self, output_type):
     return getattr(self._options, output_type, self.verbose)

For general cases, the verbose attribute could be used. For special cases, an output type name could be passed in retrieve the more fine grained setting. If the specific config is not set, it will default to the normal verbose level. Once this is in place, it would be easier to add more cases in the future if they come up.

Implementation

  • I should be able to work on an actual implementation. I'd likely start with only implementing diff verbosity since the PR might get large if I attempted to address all of the identified cases at once.
  • That being said, I think it makes sense to decide on a naming convention for these configuration fields. verbosity_{OUTPUT_TYPE} seems reasonable to me.
  • I'm leaning towards not providing CLI access to these configuration fields. They feel like they wouldn't be used regularly and it should be easy to add them after the fact if needed.
@Zac-HD Zac-HD added type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature topic: reporting related to terminal output and user-facing messages and errors labels Sep 7, 2023
@plannigan
Copy link
Contributor Author

plannigan commented Sep 16, 2023

Started working on this and unsurprisingly realized that how config.options.verbose is used currently is a bit complicated. So I took the time to see all the places that it is used. The following was based off 6c2feb75d (current head of main):

General Summary

The vast majority of uses of the verbose setting are within _pytest.terminal. In addition to using it directly, TerminalReporter also re-exposes it as a property. The pytest code base does not use the property outside of the TerminalReporter class.

  • TerminalReporter is exposed to
    plugins as part of the pytest_terminal_summary() hook.
  • The Config object is also exposed to plugins: Either directly in a hook (example) or indirectly (example)

Uses of config.options.verbose outside of TerminalReporter

Uses of config.getoption("verbose")

Uses of config.options.verbose in TerminalReporter

  • pytest_runtest_logreport():
    • <2 - possibly trimmed skipped reason
    • 2>= full skipped reason
  • pytest_collection():
    • if atty terminal and >= 0 write "collecting..."
    • if >=1 write "collecting..."
  • report_collect(): <0 exit early instead of writing summary count line
  • pytest_collection_finish(): if --collect-only passed and the session has item, > -1 write empty line before printing collected items
  • _printcollecteditems(): (only called if --collect-only)
    • < 0:
      • if < -1 write out summary of items (filename & count)
      • else write out each collected item. (filename & item name)
    • else: tree view of each item
      • >= 1: include docstring

Uses of the TerminalReporter.verbosity property

A read-only property exposed by TerminalReporter that exposes config.options.verbose

Others

  • PYTEST_DEBUG - When set, pytest will print tracing and debug information.
  • --no-header & --no-summary: explicitly disable to named terminal output component.

@plannigan
Copy link
Contributor Author

plannigan commented Sep 21, 2023

That reference is helpful to understand what things accessing verbosity and how. But it's not the easiest to understand the full behavior. So I went through and documented the behavior by verbosity level. (This helped me get a better understanding of specific aspects that can be controlled.)

-2

  • assertions
    • truncate full explanation (if not in CI)
    • abbreviate left & right repr
    • string compares - trim long sequences of leading or trailing identical characters
    • data object compares - omit identical items
    • dict compares - omit identical items
    • iterable compares - diff pretty print of objects (if in CI)
  • filter tracebacks
  • rewrite maxsize = DEFAULT_REPR_MAX_SIZE
  • (if --collect-only) write filenames and count of items in file
  • write result letter instead of item location

-1

  • assertions
    • truncate full explanation (if not in CI)
    • abbreviate left & right repr
    • string compares - trim long sequences of leading or trailing identical characters
    • data object compares - omit identical items
    • dict compares - omit identical items
    • iterable compares - diff pretty print of objects (if in CI)
  • filter tracebacks
  • rewrite maxsize = DEFAULT_REPR_MAX_SIZE
  • (if --collect-only) write each item (filename and item name)
  • write result letter instead of item location
  • write summary stats

0

  • assertions
    • truncate full explanation (if not in CI)
    • abbreviate left & right repr
    • string compares - trim long sequences of leading or trailing identical characters
    • data object compares - omit identical items
    • dict compares - omit identical items
    • iterable compares - diff pretty print of objects (if in CI)
  • write cache report (if active)
  • write stepwise report (if active)
  • filter tracebacks
  • rewrite maxsize = DEFAULT_REPR_MAX_SIZE
  • (if atty terminal) write "collecting..."
  • write collection report
  • (if --collect-only) write tree view of each item
  • write result letter instead of item location
  • write line ending progress info (if showing progress info)
  • write file path for module once before result letters for test cases in module
  • write summary stats with full width separator

1

  • write cache dir
  • write executable path
  • assertions
    • truncate full explanation (if not in CI)
    • abbreviate left & right repr
    • string compares - trim long sequences of leading or trailing identical characters
    • data object compares - omit identical items
    • dict compares - omit identical items
    • iterable compares - diff pretty print of objects
  • write cache report (if active)
  • write stepwise report (if active)
  • filter tracebacks
  • rewrite maxsize = DEFAULT_REPR_MAX_SIZE * 10
  • write "collecting..."
  • write collection report
  • (if --collect-only) write tree view of each item with docstring
  • write item location for each item
  • write trimmed (if necessary) skip reason
  • write summary stats with full width separator

2

  • write cache dir
  • write executable path
  • assertions
    • full left & right repr
    • string compares - no trimming
    • data object compares - all items listed
    • dict compares - all items listed
    • iterable compares - diff pretty print of objects
  • write cache report (if active)
  • write stepwise report (if active)
  • no rewrite maxsize
  • write "collecting..."
  • write collection report
  • (if --collect-only) write tree view of each item with docstring
  • write item location for each item
  • write full skip reason
  • write summary stats with full width separator

@plannigan
Copy link
Contributor Author

I completely missed assertions.util on the first pass (oops). I updated the previous comments with the missing details.

@The-Compiler
Copy link
Member

Should this really have been closed, since (from what I can gather) only a small part of the proposal is implemented so far?

@nicoddemus
Copy link
Member

True!

@nicoddemus nicoddemus reopened this Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: reporting related to terminal output and user-facing messages and errors type: proposal proposal for a new feature, often to gather opinions or design the API around the new feature
Projects
None yet
Development

No branches or pull requests

4 participants