Test results not updated if test is run via codelens

[Flamefire]

Environment data

• VS Code version: 1.36
• Extension version (available under the Extensions sidebar): 2019.6.24221
• OS and version: Linux Mint 19.1
• Python version (& distribution if applicable, e.g. Anaconda): Python 3.6.0
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): None
• Relevant/affected Python packages and their versions: pytest-3.9.1, same with pytest 5.0.1
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): not jedi

Expected behaviour

After fixing a test case and running it via the codelens "Run Test" the success icon in the codelens and the error squiggle on the function should reflect that result

Actual behaviour

Running a test removes the success/failure icon in front of the codelens. Error squiggles are added if the test fails, but not removed, when it succeeds.

Steps to reproduce:

1. create test function (e.g. def test_foo(): assert True)
2. Use "Run All Tests" -> tick mark appears in code lens
3. change to assert False. Run with codelens "Run Test" -> Tick dissappears, error squiggles appear
4. change back to assert True and run with codelens -> Output shows success, no tick, error squiggles persist
5. Run with "Run All Tests" -> tick mark appears in code lens, error squiggles removed

Logs

Logs look good, e.g. Python Test Log has =========================== 1 passed in 0.06 seconds ===========================

[DonJayamanne]

Thanks for reporting this issue, please could you:

• Send the contents of your settings.json (I'm interested in see the test framework you are using - unittest, nose or pytest)
• Send the output from the Test Log (please send everything)
• Finally, please could you send a screenshot of the issue.

[Flamefire]

I'm using pytest with versions above in pyenv, no other related options used

Test output does not show anything unexpected, just test passed/failed as when running pytest directly

Will add files once I'm back at the machine

[Flamefire]

Settings.json: (grep python)

"python.formatting.provider": "yapf",
"python.jediMemoryLimit": -1,
"python.testing.pytestEnabled": true,
"python.testing.unittestEnabled": false,
"python.testing.nosetestsEnabled": false,
"python.testing.pytestArgs": ["-c", "/dev/null"],
"python.jediEnabled": false,
"python.pythonPath": "/home/alex/.pyenv/shims/python",
"python.testing.cwd": "",

Test.log

python /home/alex/.vscode/extensions/ms-python.python-2019.8.29288/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear -c /dev/null
python /home/alex/.vscode/extensions/ms-python.python-2019.8.29288/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear -c /dev/null
python /home/alex/.vscode/extensions/ms-python.python-2019.8.29288/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear
============================= test session starts ==============================
platform linux -- Python 3.6.0, pytest-5.0.1, py-1.7.0, pluggy-0.12.0
Using --randomly-seed=1565185563
rootdir: /home/alex/taurushome/io-500-tests
plugins: console-scripts-0.1.7, repeat-0.7.0, randomly-1.2.3, nbval-0.9.1, cov-2.7.1, profiling-1.3.0
collected 1 item

tests/test_run.py .                                                      [100%]

-------------- generated xml file: /tmp/tmp-16882srqMED1q0pzx.xml --------------
=========================== 1 passed in 0.15 seconds ===========================

Screenshort after clicking "Run Test":

Cause

I found the issue to be solved when removing the -c /dev/null from pytestArgs. I was using this to force vanilla pytest which is not affected by e.g. my users .pytestrc (or what that file is called)

Somehow this seems to interfere with VSC test result discovery.

[DonJayamanne]

I can replicate this at my end.
For some reason the discovered tests are being reported as existing in the root directory and not under tests folder.

[bersbersbers]

I see a similar issue with nose, without any extra arguments and regardless of the way I start the tests (side bar, run all tests, ...). python.testing.pytestArgs is not set.

If this issue is more general, could the issue description be changed, e.g., to "Test results not updated if tests reside in subdirectory"?

[brettcannon]

The traditional codelenses are gone in the future test provider UI, but I can verify that refreshing from the gutter works.