What is the minimum amount of pytest's CLI that should be supported by integrating editors?

[brettcannon]

Slightly odd question, but what I'm wondering is if an editor were to integrate with pytest, what is the minimal amount of pytest's CLI would have to be supported? If we were to tell users that they were expected to use pytest.ini and such as much as possible, what would still need to be supported via settings in order for pytest to work?

Assume we know the directory that is open in the editor. Would specifying a different working directory be needed? What about the location of a pytest.ini file? Neither of these? Something I'm not thinking of?

Or a slightly different phrasing is what is the minimal amount of custom UI an editor would need to provide to run pytest (assume #8458 takes care of discovery/execution UX; this is about running pytest)?

[nicoddemus]

Hey @brettcannon

what is the minimal amount of pytest's CLI would have to be supported?

As a user, I kind expected to be able to pass any command-line to pytest when executing it inside my editor/IDE. For example, PyCharm has a free form field "Additional arguments" where I can pass any flag I want.

Many flags require command-line arguments which are not expected to be written "permanently" in a the configuration file, but should be used on occasion, for example --last-failed or --stepwise. The same is true for external plugins.

Assume we know the directory that is open in the editor. Would specifying a different working directory be needed? What about the location of a pytest.ini file? Neither of these? Something I'm not thinking of?

I'm might be biased, but both PyCharm and PyDev (IIRC about the latter) execute pytest from the directory of the current file, and let pytest itself find the configuration file. I particularly like that default.

Or a slightly different phrasing is what is the minimal amount of custom UI an editor would need to provide to run pytest

TBH I think an "Additional parameters" is all that's needed, but I might be having tunnel vision because I'm used to having that available to me in PyCharm.

[brettcannon]

Depends on the sets of tests being executed. If you are executing the tests from a single test file, set the current working directory to the directory of the test file, if it is a directory containing several test files, set to that directory, and so on.

So I'm thinking of the bare pytest scenario where pytest traverses down, collects everything, and starts running tests. Now let's say that the tests are all in a tests/ directory. Would that mean you expect the working directory to be where you executed pytest from or tests/? It seems like the latter, but I want to make sure I'm understanding clearly.

But this reminds me of an awesome feature that I often miss, where the editor is smart enough to detect the test file associated with a module/package, and have a shortcut to execute those tests.

You are definitely not the first to ask for this. 😅

we search for the config file based on the tests given in the command-line, not on the current working directory (except when you don't pass any test file or directory, in which case we use the current working directory to find the config file).

One interesting thing I just realized is how this will muck with relative file paths in tests (e.g. open() to load some golden file). Might still need to provide a way for people to override where they want to anchor their execution for the working directory but default to the semantics you're suggesting.

[nicoddemus]

It seems like the latter, but I want to make sure I'm understanding clearly.

I agree. However this is a convention decided by the IDE. I would also understand/accept if the IDE decides to always run the tests using the directory containing pytest.ini as current working directory, for example.

You are definitely not the first to ask for this. 😅

Heh!

One interesting thing I just realized is how this will muck with relative file paths in tests (e.g. open() to load some golden file).

Definitely. While the IDE should have an option to control this, I think tests that depend on the current working directory to work are an anti-pattern, users should be getting relative paths based on __file__, or providing those golden file paths through a fixture (which would also use __file__ to locate it). That's just a comment of course, the IDE can't prevent those anti-patterns so it should at least have clear rules to which current working directory it will use to run tests.

[brettcannon]

I would also understand/accept if the IDE decides to always run the tests using the directory containing pytest.ini as current working directory, for example.

Is there a way that a plug-in gets told where the config file lives? I quick check of the docs didn't yield anything obvious for me.

While the IDE should have an option to control this, I think tests that depend on the current working directory to work are an anti-pattern

Totally agree! I just know I have answered my fair share of Stack Overflow questions caused by this sort of thing. 😄

[RonnyPfannschmidt]

@brettcannon there is config.inipath/inifile for the ini config, and there is the tool.pytest section for pyproject.toml

[brettcannon]

@RonnyPfannschmidt Thanks! Found https://docs.pytest.org/en/stable/reference.html#pytest.config.Config.inipath.