Unittest: Only the last TestCase in alphabetical order was executed

[MmAaXx500]

Environment data

• VS Code version: 1.51.1
• Extension version (available under the Extensions sidebar): 2020.11.358366026
• OS and version: Windows 10 Ver: 20H2 Build: 19042.630
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.6 64-bit
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): with and without venv
• Relevant/affected Python packages and their versions: XXX
• Relevant/affected Python-related VS Code extensions and their versions: XXX
• Value of the python.languageServer setting: Jedi (default)

Expected behaviour

The expected behaviour is if I click on the "Run Current Test File" all test classes and functions are executed.

Actual behaviour

The actual behaviour is only the last TestCase in alphabetically order and its functions are executed (Run all and one-by-one classes are ok).

Steps to reproduce:

(Here is a complete workspace)

1. Create an empty workspace with Python extension enabled
2. Insert the following in "test.py":

import unittest

class TestsA(unittest.TestCase):
    def testA1(self):
        assert True

class TestsB(unittest.TestCase):
    def testB1(self):
        assert True

3. Run all test in the file with "Run Current Test File"

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Log of test discovery and running one file (without venv):

> ~\AppData\Local\Programs\Python\Python38\python.exe -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover("./mapps", pattern="test*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: e:\test
> ~\AppData\Local\Programs\Python\Python38\python.exe -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover("./mapps", pattern="test*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: e:\test
Error 2020-11-14 16:15:30: stderr jediProxy Error (stderr) c:\Users\Max\.vscode\extensions\ms-python.python-2020.11.358366026\pythonFiles\completion.py:584: DeprecationWarning: Deprecated since version 0.16.0. Use Script(...).get_names instead.
  jedi.api.names(

> ~\AppData\Local\Programs\Python\Python38\python.exe c:\Users\Max\.vscode\extensions\ms-python.python-2020.11.358366026\pythonFiles\visualstudio_py_testlauncher.py --us=./mapps --up=test*.py --uvInt=2 --result-port=60721 -ttest --testFile=e:\test\mapps\test.py
cwd: e:\test
> ~\AppData\Local\Programs\Python\Python38\python.exe c:\Users\Max\.vscode\extensions\ms-python.python-2020.11.358366026\pythonFiles\visualstudio_py_testlauncher.py --us=./mapps --up=test*.py --uvInt=2 --result-port=60721 -ttest --testFile=e:\test\mapps\test.py
cwd: e:\test
Error 2020-11-14 16:15:41: Error: read ECONNRESET 
Error 2020-11-14 16:15:42: stderr jediProxy Error (stderr) c:\Users\Max\.vscode\extensions\ms-python.python-2020.11.358366026\pythonFiles\completion.py:584: DeprecationWarning: Deprecated since version 0.16.0. Use Script(...).get_names instead.
  jedi.api.names(

Log of test discovery and running one file (with venv):

> e:\test\venv\Scripts\python.exe -c "
import unittest
loader = unittest.TestLoader()
suites = loader.discover("./mapps", pattern="test*.py")
print("start") #Don't remove this line
for suite in suites._tests:
    for cls in suite._tests:
        try:
            for m in cls._tests:
                print(m.id())
        except:
            pass"
cwd: e:\test
> e:\test\venv\Scripts\python.exe c:\Users\Max\.vscode\extensions\ms-python.python-2020.11.358366026\pythonFiles\visualstudio_py_testlauncher.py --us=./mapps --up=test*.py --uvInt=2 --result-port=60788 -ttest --testFile=e:\test\mapps\test.py
cwd: e:\test
Error 2020-11-14 16:19:12: Error: read ECONNRESET 

[ericsnowcurrently]

Thanks for the bug report! We investigate issues in order based on priority and severity, which includes the impact it has on your ability to use the extension to do productive work, and the number of people affected. If other users come forward and leave a comment demonstrating they are seeing/reproducing the problem then we will raise this issue's priority.

If you think your issue is more of a question or configuration problem rather than a bug, please ask on Stack Overflow with the visual-studio-code and python labels for help.

Thanks for your understanding and patience!

[spapadim]

I have encountered the same error. After a bit of digging around, it seems that the cause is in pythonFiles/visualstudio_py_testlauncher.py, which, starting at line 336, does:

for test_suite in suites._tests:
    for cls in test_suite._tests:
        try:
            for m in cls._tests:
                testId = m.id()
                if testId.startswith(opts.tests[0]):
                    suite = cls
                if testId == opts.tests[0]:
                    tests = unittest.TestSuite([m])
                    break
        except Exception as err:
            errorMessage = traceback.format_exception()
            pass
if tests is None:
  tests = suite

Each suite = cls assignment overwrites the previous, so only the last test suite (i.e., in this case, the last TestCase class in a file with more than one) remains. If, instead, this is changed to:

tests = unittest.TestSuite()
for test_suite in suites._tests:
    ....
                if testId.startswith(opts.tests[0]):
                    tests.addTest(cls)
                    break
    ....

this seems to resolve the issue. Presumably, the check tests is None or suites is None immediately below (to issue a "Failed to identify error") should also be replaced with something like tests is None or tests.countTests() == 0 (although not 100% sure this is entirely equivalent to prior behavior).

The only issue I see still remaining is that if

• MyTestCase derives from base class CommonTestCase with
  • CommonTestCase is a subclass of TestCase (i.e., not a mixin), and
  • CommonTestCase itself defines methods starting with test_, but
• MyTestCase itself defines no test_ methods,
  then the MyTestCase suite would be missed. I think this is a rare enough situation (who would do this??) so as to not need to worry. In that case, it's anyway possible some other parts of the UI (let alone humans :) ) also get confused.

FWIW, I'm attaching a diff of all my edits (although it might take longer to download and merge, than to re-type it.. :) ). Thanks!

visualstudio_py_testlauncher.py.diff.txt

[jeremyn]

It looks like this is the same problem as in #10972 which says we should get a fix soon.

[karthiknadig]

For anyone running into this. Please try the insiders edition of the extension to see if this is addressed.

[jeremyn]

I tried the two-class sample code from the original post in Insiders and it looks fixed, thanks.

[MmAaXx500]

Yes, it looks like it's fixed. Thanks!

[k870611]

I have the same problem even use with insiders

VS Code version: 1.60.0-insider
Extension version (available under the Extensions sidebar): v2021.8.1147840270
OS and version: Windows 10 Ver: 20H2 Build: 19042.1165
Python version (& distribution if applicable, e.g. Anaconda): 3.6.2 64-bit
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): with and without venv
Relevant/affected Python packages and their versions: XXX
Relevant/affected Python-related VS Code extensions and their versions: XXX
Value of the python.languageServer setting: Jedi (default)

Actual behaviour: Same, will run the last TestCase

error msg:

C:\Python36\python.exe c:\Users\Huang.vscode-insiders\extensions\ms-python.python-2021.8.1147840270\pythonFiles\visualstudio_py_testlauncher.py --us=./mapps --up=test*.py --uvInt=2 --result-port=7006 -ttest --testFile=c:\Users\Huang\Downloads\test-workspace\mapps\test.py
cwd: c:\Users\Huang\Downloads\test-workspace
C:\Python36\python.exe c:\Users\Huang.vscode-insiders\extensions\ms-python.python-2021.8.1147840270\pythonFiles\visualstudio_py_testlauncher.py --us=./mapps --up=test*.py --uvInt=2 --result-port=7006 -ttest --testFile=c:\Users\Huang\Downloads\test-workspace\mapps\test.py
cwd: c:\Users\Huang\Downloads\test-workspace
Error 2021-08-21 10:05:53: Error: read ECONNRESET

[karthiknadig]

@k870611 I meant insider version of Python extension (it should be 2021.9.*) you can download it here https://pvsc.blob.core.windows.net/extension-builds/ms-python-insiders.vsix

[k870611]

@karthiknadig

thanks, it work now.

[k870611]

I have found a bug of this version.

I will create log to recode test message and it will be created for each test ( if I write 10 test in py it will create 10 log file) but in official release 2021.8.1147840270 it will create once ( even I create 10 test in py it will only create once and recode all test message into log)

FYR