Rewrite testing adapter

[kimadeline]

Meta-issue: #16936

Unittest

• Test discovery with unittest - Unittest discovery using the new test adapter #18253
• Test execution with unittest - Refactor unittest test execution #19098
• Test debugging with unittest

Pytest

• Prototyping pytest plugin Prototype pytest plugin for test discovery/execution. #19790
• Add pytest plugin Review and Commit Pytest Plugin (without enabling) #20077
• Test discovery with pytest
  • Write discovery logic for pytest plugin Finalize Pytest Discovery Logic  #20078
  • Display discovered test on UI Connect Pytest Discovery Plugin to Testing UI #20079
• Test execution with pytest
  • Create pytest run tests call Create Pytest Plugin Call for Run Test #20080
  • Write run tests plugin logic Pytest Run Tests Logic #20081
  • Display run test info on UI Surface Test Results to UI #20082
• Pytest error handling Error Handling Pytest Discovery #20083
• Complete pytest testing suite Write testing for pytest discovery #20085
• Enable pytest rewrite to users Enable Pytest Rewrite to Users #20086
• Test debugging with pytest

Cleanup

• Clean up unittest-related issues
• Clean up pytest-related issues
• Make sure there are tests for the test controller after the rewrite

Notes

Public issues

1. python.testing.cwd ignored when discovering or running tests #8678
2. Tests not discovered, but running tests anyway works fine. #16565
3. Handle exclude lists in a test run request #16929
4. Go to Test doesn't work for test class items in Test Explorer #17107
5. Tests are discovered correctly but then an incorrect argument is supplied to PyTest, leading test-result discovery to fail (Git Submodules?) #17903
6. Unittest discovery not working when sys.stdout is set in global scope #17957

Issues we expect this rewrite will fix

• Problem with test IDs
  • Folders named 'py' result in "Test result not found" #17414
  • VSCode not running discovered PyTest tests when path contains a symlink #17573
  • "Test result not found" if cwd is a subdirectory in the workspace #18030
  • Escape test ID`s in Testing API #17810
• Problems with other plugins or python or libs printing to output
  • Pytest discovery stuck in Output panel #18656
• Problems with test discovery
  • pytest discovery error #19809
  • Improve python test discovery error reporting #19978
  • Group test explorer test by workspace in multi-root workspace #20345
  • unexpected situation while processing a pytest node during test discovery #19804
  • Test discovery should be fault tolerant and display all successfully imported files #13301
• Problems with testing output
  • Debug output for test discovery has misleading argument #19820
  • Support an output stream from the testing adapter script. #6594
• Problems with remote setups
  • After switching to devcontainer (based on 3.10-bullseye python) can't go to test or run test #19857
  • Pytest test discovery over SSH does not work #19864
• Problems with testing args
  • PyTest optional args don't support wildcards #19918
  • Having PYTHONWARNINGS=default crashes pytest discovery #20259
  • Python unittest runner doesn't respect arguments to pass to unittest runner #20425
• Performance Concerns
  • Testing: pytest should be running multiple nodes in one single process #19796
• Problems with Running Tests
  • Use pytest hooks for test execution instead of "scraping" the JUnit XML results. #12973

pytest feedback on how to things the "right way"

• What is the preferred way for editors/tools to interact with pytest? pytest-dev/pytest#8458
• What is the minimum amount of pytest's CLI that should be supported by integrating editors? pytest-dev/pytest#8506
• visualstudio_py_testlauncher.py - format_exception() missing 3 required positional arguments: 'etype', 'value', and 'tb' #3305
• Red test failure output in editor gets wrapped over lines #18047

unittest missing features

• support for subtests status unittest runner thinks failing test is not run #17561
• Unittest tests will not run with unittest.main() in the test file #17342
• should_never_reach_here for particular test case when parametrized test has unmatched brackets or parenthesis #17676

[Bilalh]

It would be nice if other python testing frameworks such as behave (a BDD framework) https://behave.readthedocs.io/en/stable/index.html were supported by the testing adapter.

[kimadeline]

Hi @Bilalh, support for behave is not within scope of this issue. Feel free to open a feature request on the repo and we'll take a look at it. Thanks!

[eleanorjboyd]

Hello everyone! We are wrapping up the rewrite of the testing infrastructure and heading into the stage of testing issues and seeing which ones remain. Currently, we are rolling out this new infrastructure in parallel and users can try out the new changes by setting "python.experiments.optInto": ["pythonTestAdapter"] in their user settings.json. Right now you must be on vscode insiders to test but we are in the process of switching all users to the rewrite incrementally so if you do not have insiders, watch our release notes to get updated on when it will begin to hit stable.

I will be triaging our backlog of testing issues and determining which issues still exist. In the meantime, if you have an issue that was waiting on the rewrite to potentially be fixed now is a great time to test it! If you do give it a try, please update the specific issue with the outcome (success or how it is still failing) so I can more quickly determine the ongoing issues following the rewrite. Thanks everyone for your interest and patience as we get this rolled out!

[JensMadsen]

"python.experiments.optInto": ["pythonTestAdapter"]

@eleanorjboyd I have enabled the pythonTestAdapter. I still see that the startup time is very long. I use unittest. When I run a single unit test stuff related to tests elsewhere (e.g. other folders) is initialized. Example: I run a straightforward test that tests some simple string gymnastics. Before the test starts I see in the output terminal that e.g. Pytorch is imported which takes a long time.

Please let me know if I can help in any way debugging this :-)

[eleanorjboyd]

Hi @JensMadsen! Could you open a new issue and include some of these logs? From there I can help you out a bit more- thanks!

[JensMadsen]

Hi @JensMadsen! Could you open a new issue and include some of these logs? From there I can help you out a bit more- thanks!

@eleanorjboyd I will do that but I am unsure what logs you refer to.

[eleanorjboyd]

If you go to the command palette and look for the command- python: report issue... that is the best way submit all the logs and info to us as a new issue so I can take a look. Hopefully that would work- thanks!

[JensMadsen]

@eleanorjboyd done #24022