Nose Tests discovery fails when tests are split by folders and imported to __init__.py (wantModule and wantDirectory are not parsed)

[IBestuzhev]

Environment data

• VS Code version: 1.37.1
• Extension version (available under the Extensions sidebar): 2019.8.30787
• OS and version: Windows 10 running VS Code, remote plugin connects to Ubuntu 18.04.2 via SSH
• Python version (& distribution if applicable, e.g. Anaconda): python 2.7.15rc1
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: Django==1.11.23; django-nose==1.4.6; nose==1.3.7
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): jedi

Expected behaviour

Test discovery for nose should scan for both wantFile and wantModule and/or wantDirectory log outputs

Actual behaviour

We have a django project with django-nose test runner. Tests are split into modules, and all tests are imported to __init__.py. I think these are some obsolete tests, from default django test runner. But nose runs them perfectly.

I create a simple script, that translates the nosetests call to ./manage.py test call.
Basically it just strips -vvv and replaces it with --verbosity

#!/bin/bash
function arg_remove ()
{
    local array=("${@:3}")

    for ((i=0; i<"${#array[@]}"; ++i)); do
        case ${array[i]} in
            "$2") unset array[i]; break ;;
        esac
    done

    # clean up unset array indexes
    for i in "${!array[@]}"; do
        new_array+=( "${array[i]}" )
    done
    array=("${new_array[@]}")
    unset new_array

    # assign array outside function scope
    local -g "$1=(  )"
        eval ${1}='("${array[@]}")'
}
arg_remove ARG_PASS "-vvv" "$@"
echo $@
direnv exec /home/igor/myproject/backend/ /home/igor/.virtualenvs/myproject/bin/python /home/igor/myproject/backend/manage.py test --settings=core.local_settings --verbosity=3 -l nose.selector --nologcapture "${ARG_PASS[@]}"

The structure of tests is (stripped)

File backend/apps/tests/__init__.py

from .action import ActionTestCase

File backend/apps/tests/action.py

from apps.myproj.tests.base import BaseTestCase


class ActionMSFTestCase(BaseTestCase):
    pass
    # stripped

The output of --collect-only command is different to what vscode-python expects.

Relevant output to tests above is:

nose.selector: DEBUG: wantDirectory /home/igor/ata_portal/backend/apps/api/tests? True
nose.selector: DEBUG: Test name /home/igor/ata_portal/backend/apps/api/tests resolved to file /home/igor/ata_portal/backend/apps/api/tests, module None, call None
nose.selector: DEBUG: Final resolution of test name /home/igor/ata_portal/backend/apps/api/tests: file /home/igor/ata_portal/backend/apps/api/tests module apps.api.tests call None
nose.selector: DEBUG: wantModule <module 'apps.api.tests' from '/home/igor/ata_portal/backend/apps/api/tests/__init__.pyc'>? True
nose.selector: DEBUG: wantClass <class 'apps.api.tests.action_msf.ActionMSFTestCase'>? True

parserService here scans only for wantFile output.

In case above it founds wantClass above any wantFile ... py? True line. That causes testFile.suites.push(testSuite); to fail

Steps to reproduce:

1. Create a package tests
2. Add a module action.py to package with some real test case
3. import action inside tests/__init__.py
4. Configure nose tests for workspace
5. Run Python: Discover Tests command

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Test Discovery failed: 
TypeError: Cannot read property 'suites' of undefined

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

notificationsAlerts.ts:40 Test discovery error, please check the configuration settings for the tests.
onDidNotificationChange @ notificationsAlerts.ts:40
console.ts:137 [Extension Host] Python Extension: displayDiscoverStatus TypeError: Cannot read property 'suites' of undefined
	at t.forEach.t (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1015174)
	at Array.forEach (<anonymous>)
	at h.parseNoseTestModuleCollectionResult (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1014343)
	at e.split.forEach (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1013968)
	at Array.forEach (<anonymous>)
	at h.getTestFiles (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1013849)
	at h.parse (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1013661)
	at h.discoverTests (/home/igor/.vscode-server/extensions/ms-python.python-2019.8.30787/out/client/extension.js:75:1012881)
	at process._tickCallback (internal/process/next_tick.js:68:7)

[ericsnowcurrently]

Thanks for letting us know about this, @IBestuzhev. This sounds similar to a problem with the latest version of pytest (#6990), where the XML output changed. We use JUnit XML files to get the output from nose. Was this working on an earlier version of nose? If so then we can look into that further.

Just in case that isn't the problem, I have some questions that may help us figure out what is going on:

• what version of nose are you using? (Presumably it is nose2 since nose "1" is effectively EOL.)
• what happens when you run the nose directly in the terminal?
• what is the command (with args) that you use to do so?
• how is the bash script you included above related to running the tests?
• what is in your workspace settings.json?

Also note that our support for nose is fairly low, due to lack of demand. However, that doesn't mean we won't try to help you resolve this issue. :)

[IBestuzhev]

Thanks @ericsnowcurrently

Yes, I understand it's quite rare issue, the project is legacy and so it uses not recent versions.

Was this working on an earlier version of nose?

Never checked before.

what version of nose are you using?

nose==1.3.7

what happens when you run the nose directly in the terminal?
what is the command (with args) that you use to do so?

That is Django project, so I can't run nosetests command directly. It fails to import most of the tests and they require django.setup() to be called.

The error is AppRegistryNotReady: Apps aren't loaded yet. And I think it's not relevant to this issue.

But I can run python manage.py test, and this command does some django setup and then runs nosetests --verbosity=2 --with-id

So for test discovery I run ./manage.py test --settings=core.local_settings --verbosity=3 -l nose.selector --nologcapture --collect-only

how is the bash script you included above related to running the tests?

It translates nosetests call to python manage.py test command

what is in your workspace settings.json?

"python.testing.cwd": "/home/igor/project/backend",
"python.testing.nosetestPath": "/home/igor/custom_nose",  # custom bash script
"python.testing.promptToConfigure": false,

[ericsnowcurrently]

@IBestuzhev, we've added a fix for our JUnit XML parsing. Please see if that fixes the problem. It is available through the "insiders" build of the extension (set the "python.insidersChannel" in your user settings to "daily").

[IBestuzhev]

Hi @ericsnowcurrently
Thanks for letting me know.
I downloaded insiders build and run test discovery.

Nothing changed.

Also, my initial problem was with parsing of Nose output, not JUnit output.

[ericsnowcurrently]

@IBestuzhev, sorry for the confusion. I was able to reproduce the problem using the steps you originally provided. It looks like we expect a file to be picked if a class is picked, but that isn't the case here. In my nose output I see the following:

...
nose.loader: DEBUG: Load from module <module 'tests' from '/.../gh-7124/tests/__init__.py'>
nose.selector: DEBUG: wantClass <class 'tests.action.ActionTestCase'>? True
...
nose.selector: DEBUG: wantFile /.../gh-7124/tests/action.py? None
...

We'll investigate further.

[ericsnowcurrently]

My full nose output:

$ /usr/bin/python3.7 -m nose --collect-only -vvv tests
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.plugins.manager: DEBUG: Configuring plugins
nose.plugins.manager: DEBUG: Plugins enabled: [<nose.plugins.capture.Capture object at 0x7f09e6b18748>, <nose.plugins.logcapture.LogCapture object at 0x7f09e6bb30b8>, <nose.plugins.deprecated.Deprecated object at 0x7f09e6b4ef60>, <nose.plugins.skip.Skip object at 0x7f09e68ab4e0>, <nose.plugins.collect.CollectOnly object at 0x7f09e68ab5f8>]
nose.core: DEBUG: configured Config(addPaths=True, args=(), configSection='nosetests', debug=None, debugLog=None, env={}, exclude=None, files=[], firstPackageWins=False, getTestCaseNamesCompat=False, ignoreFiles=[re.compile('^\\.'), re.compile('^_'), re.compile('^setup\\.py$')], ignoreFilesDefaultStrings=['^\\.', '^_', '^setup\\.py$'], include=None, includeExe=False, logStream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, loggingConfig=None, options=<Values at 0x7f09e631a208: {'version': False, 'showPlugins': False, 'verbosity': 4, 'files': None, 'where': None, 'py3where': None, 'testMatch': '(?:^|[\\b_\\./-])[Tt]est', 'testNames': None, 'debug': None, 'debugLog': None, 'loggingConfig': None, 'ignoreFiles': [], 'exclude': [], 'include': [], 'stopOnError': False, 'addPaths': True, 'includeExe': False, 'traverseNamespace': False, 'firstPackageWins': False, 'byteCompile': True, 'attr': None, 'eval_attr': None, 'capture': True, 'logcapture': True, 'logcapture_format': '%(name)s: %(levelname)s: %(message)s', 'logcapture_datefmt': None, 'logcapture_filters': None, 'logcapture_clear': False, 'logcapture_level': 'NOTSET', 'enable_plugin_coverage': None, 'cover_packages': None, 'cover_erase': None, 'cover_tests': None, 'cover_min_percentage': None, 'cover_inclusive': None, 'cover_html': None, 'cover_html_dir': 'cover', 'cover_branches': None, 'cover_xml': None, 'cover_xml_file': 'coverage.xml', 'debugBoth': False, 'debugFailures': False, 'debugErrors': False, 'noDeprecated': False, 'enable_plugin_doctest': None, 'doctest_tests': None, 'doctestExtension': None, 'doctest_result_var': None, 'doctestFixtures': None, 'doctestOptions': None, 'enable_plugin_isolation': None, 'detailedErrors': None, 'noSkip': False, 'enable_plugin_id': None, 'testIdFile': '.noseids', 'failed': False, 'multiprocess_workers': 0, 'multiprocess_timeout': 10, 'multiprocess_restartworker': False, 'enable_plugin_xunit': None, 'xunit_file': 'nosetests.xml', 'xunit_testsuite_name': 'nosetests', 'enable_plugin_allmodules': None, 'collect_only': True}>, parser=<optparse.OptionParser object at 0x7f09e837e7b8>, parserClass=<class 'optparse.OptionParser'>, plugins=<nose.plugins.manager.DefaultPluginManager object at 0x7f09e837e3c8>, py3where=(), runOnInit=True, srcDirs=('lib', 'src'), stopOnError=False, stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, testMatch=re.compile('(?:^|[\\b_\\./-])[Tt]est'), testMatchPat='(?:^|[\\b_\\./-])[Tt]est', testNames=['tests'], traverseNamespace=False, verbosity=4, where=(), worker=False, workingDir='/.../gh-7124')
nose.importer: DEBUG: Add path /.../gh-7124
nose.plugins.collect: DEBUG: Preparing test loader
nose.core: DEBUG: test loader is <nose.loader.TestLoader object at 0x7f09e631a358>
nose.core: DEBUG: defaultTest .
nose.core: DEBUG: Test names are ['tests']
nose.core: DEBUG: createTests called with None
nose.loader: DEBUG: load from tests (None)
nose.selector: DEBUG: Test name tests resolved to file tests, module None, call None
nose.selector: DEBUG: Final resolution of test name tests: file /.../gh-7124/tests module tests call None
nose.importer: DEBUG: Import tests from /.../gh-7124
nose.importer: DEBUG: Add path /.../gh-7124
nose.importer: DEBUG: find module part tests (tests) in ['/.../gh-7124']
nose.loader: DEBUG: Load from module <module 'tests' from '/.../gh-7124/tests/__init__.py'>
nose.selector: DEBUG: wantClass <class 'tests.action.ActionTestCase'>? True
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.addCleanup>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.addTypeEqualityFunc>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertAlmostEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertCountEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertDictContainsSubset>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertDictEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertFalse>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertGreater>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertGreaterEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIn>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIs>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsInstance>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsNone>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsNot>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertIsNotNone>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertLess>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertLessEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertListEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertLogs>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertMultiLineEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotAlmostEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotIn>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotIsInstance>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertNotRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertRaises>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertRaisesRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertSequenceEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertSetEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertTrue>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertTupleEqual>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertWarns>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.assertWarnsRegex>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.countTestCases>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.debug>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.defaultTestResult>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.doCleanups>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.fail>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.deprecated_func>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.id>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.run>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.setUp>? None
nose.selector: DEBUG: wantMethod <bound method TestCase.setUpClass of <class 'tests.transplant_class.<locals>.C'>>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.shortDescription>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.skipTest>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.subTest>? None
nose.selector: DEBUG: wantMethod <unbound method ActionTestCase.tearDown>? None
nose.selector: DEBUG: wantMethod <bound method TestCase.tearDownClass of <class 'tests.transplant_class.<locals>.C'>>? None
nose.plugins.collect: DEBUG: TestSuite(<map object at 0x7f09e631a9b0>)
nose.loader: DEBUG: Load tests from module path /.../gh-7124/tests?
nose.loader: DEBUG: path: /.../gh-7124/tests os.path.realpath(/.../gh-7124/tests): /.../gh-7124/tests
nose.loader: DEBUG: load from dir /.../gh-7124/tests
nose.importer: DEBUG: Add path /.../gh-7124/tests
nose.importer: DEBUG: Add path /.../gh-7124
nose.selector: DEBUG: __init__.py matches ignoreFiles pattern; skipped
nose.selector: DEBUG: wantFile /.../gh-7124/tests/action.py? None
nose.plugins.collect: DEBUG: TestSuite(<nose.suite.ContextList object at 0x7f09e631a9b0>)
nose.plugins.collect: DEBUG: Add test <nose.plugins.collect.TestSuite tests=[]>
nose.plugins.collect: DEBUG: TestSuite([<nose.plugins.collect.TestSuite tests=[<nose.plugins.collect.TestSuite tests=[]>]>])
nose.plugins.collect: DEBUG: Add test <nose.plugins.collect.TestSuite tests=[<nose.plugins.collect.TestSuite tests=[]>]>
nose.core: DEBUG: runTests called

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

[kainjow]

I'm experiencing a similar (or same) issue. Also using nose 1.3.7, but on Python 3.7, macOS 10.15, and VSCode 1.45.0.

@IBestuzhev your shell script idea worked for me, although I modified it to work on macOS. Thanks!

[karthiknadig]

Closing via #16371