If multiple tests have the same name, only one is run

[Artfunkel]

Issue Type: Bug

Here is a sample test script in which the same test is executed with different configurations.

import unittest

class Foo(unittest.TestCase):
    config = "Hello"
    def test_Print(self):
        print(self.config)

class FooConfigured(Foo):
    config = "World"

if __name__ == '__main__':
    unittest.main()

If I try to run all tests in the file, only FooConfigured is executed. If I try to run the Foo class as a whole, the same thing happens. I have to explicitly run Foo.test_Print or it will never be executed.

In my real world project there are 15 test methods, and the configuration required is picking a version of a dependency to run against.

Extension version: 2019.11.50794
VS Code version: Code 1.40.1 (8795a9889db74563ddd43eb0a897a2384129a619, 2019-11-13T16:49:30.162Z)
OS version: Linux x64 4.15.0-70-generic

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-3210M CPU @ 2.50GHz (4 x 2531)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled metal: disabled_off multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: disabled_software skia_renderer: disabled_off surface_control: disabled_off surface_synchronization: enabled_on video_decode: unavailable_off viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off webgl: enabled webgl2: enabled
Load (avg)	1, 1, 0
Memory (System)	11.46GB (7.43GB free)
Process Argv	--no-sandbox --unity-launch
Screen Reader	no
VM	0%

[ericsnowcurrently]

Thanks for getting in touch with us on this, @Artfunkel. I was not able to reproduce the problem using the example code you provided. Instead, both tests were discovered and both were run. I tried it with both unittest and pytest. Here are some questions to help identify the problem:

• what do you have in your workspace settings.json?
• how are you running the tests?
• what do you see in the test explorer?
• do you see code lenses?
• what do you see in the "Python Test Log" output panel?

Thanks!

[Artfunkel]

I can reproduce this issue with a clean project, but the settings.json of my real project says this:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./Tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.unittestEnabled": true,
    "python.pythonPath": "/usr/bin/python3.7"
}

I'm running tests from the standard VS Code "Tests" panel by clicking on the green "Run" icons.

I see all of my tests listed correctly.

I see code lens on both test classes, but not on the individual methods.

After clicking the green "Run" icon on the .py file, I see this in the output log:

test_tests.Foo.test_Print
test_tests.FooConfigured.test_Print
test_Print (test_tests.FooConfigured) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OKWorld

[ericsnowcurrently]

I was able to reproduce the problem, but only in one certain case. If I run tests for the file (in test explorer or by command) then only the one test is run, even though both are discovered. If I "run all tests" then both tests get run. (I got the same behavior under unittest and pytest.)

We will look into this and work on a fix. Sorry for the inconvenience!

[ericsnowcurrently]

As to code lenses, there is no function to add one to in FooConfigured, so that is the expected behavior.

[ericsnowcurrently]

Possible causes:

• the discovered test is not getting recorded properly (relative to the file)
• the code that actually runs the tests for a file is making an invalid assumption and not handling the case (same file, same test name, same bound class, different discovered class) properly

[brettcannon]

This is fixed in our upcoming test provider.