Code coverage breaks in parallel mode

[eye1]

• Version: v14.18.2
• Platform: 19.6.0 Darwin Kernel Version 19.6.0: Sun Nov 14 19:58:51 PST 2021; root:xnu-6153.141.50~1/RELEASE_X86_64 x86_64

When adding --parallel to this command:
"test": "NODE_ENV=test mocha \"test/**/*.test.js\" --parallel --timeout 10000 --exit --require test/setup.mjs",

code coverage report seems to only receive partial data

small excerpt in a much longer report:

domain/repositories                   |   11.23 |    34.28 |    8.88 |   11.23 |                   
  applicationRepository.js             |       0 |        0 |       0 |       0 | 1-37              
  cacheRepository.js                   |   64.78 |      100 |   14.28 |   64.78 | ...53,57-59,63-65 
  ...PaymailPublicProfileRepository.js |       0 |        0 |       0 |       0 | 1-17              
  changeLogRepository.js               |       0 |        0 |       0 |       0 | 1-21              
  developerAccountRepository.js        |       0 |        0 |       0 |       0 | 1-14              
  environmentVariableRepository.js     |     100 |      100 |      50 |     100 |                   
  exchangeRateRepository.js            |   56.66 |      100 |   28.57 |   56.66 | ...16,20-35,41-43 
  fiatRampsGlobalSettingsRepository.js |   31.03 |      100 |      20 |   31.03 | 7-9,15-35,39-54   
  fiatRampsWebHookRepository.js        |       0 |        0 |       0 |       0 | 1-23              
  invoiceRepository.js                 |       0 |        0 |       0 |       0 | 1-19              
  lockRepository.js                    |       0 |        0 |       0 |       0 | 1-39              
  nonceRepository.js                   |       0 |        0 |       0 |       0 | 1-13              
  paymailWhiteListRepository.js        |       0 |        0 |       0 |       0 | 1-16              
  paymentRequestRepository.js          |       0 |        0 |       0 |       0 | 1-17              
  ...eVerificationRequestRepository.js |       0 |        0 |       0 |       0 | 1-37              
  rawTransactionRepository.js          |       0 |        0 |       0 |       0 | 1-56              
  reservedHandleRepository.js          |       0 |        0 |       0 |       0 | 1-15              
  topUpOrderRepository.js              |       0 |        0 |       0 |       0 | 1-180             
  transactionRecordRepository.js       |       0 |        0 |       0 |       0 | 1-295             
  ...dTransactionTemplateRepository.js |       0 |        0 |       0 |       0 | 1-59              
  unspentOutputRepository.js           |       0 |        0 |       0 |       0 | 1-447             
  userAddressRepository.js             |       0 |        0 |       0 |       0 | 1-64              
  userAuthorizationRepository.js       |       0 |        0 |       0 |       0 | 1-74   

"mocha": "9.1.4"

If I upgrade to mocha 9.2.0 then it does not work even without parallel and the report becomes empty.

[bcoe]

@eye1 could you provide a minimal example that fails in a repository?

c8 itself uses mocha, and is tested with mocha@9.2.2.

[bcoe]

@eye1 if you're continuing to bump into issues, and can provide an example, please let me know. Closing for now.

[kf6kjg]

I've got a similar result but with a different trigger. In my case setting mocha's --parallel flag true or false doesn't change the behavior. Instead running multiple instances of c8 in parallel with --inspect=0.0.0.0 appears to cause the behavior. When doing this it appears to be random which package will fail, likely due to a race condition on which grabs the inspection port first.

Stable workaround appears to be to remove the --inspect=0.0.0.0 flag.

In my case I've got a monorepo using NX. NX will run the test suites for each package independently and in parallel on a multicore/cpu machine. However you don't need a monorepo or NX to reproduce what I'm observing. Here I demonstrate using GNU Parallel:

parallel \
    --tag \
    '
        cd packages/{1} \
        && npx c8 \
            --config ../../.c8rc.json \
            --report-dir "../../.coverage/report/{1}" \
            --temp-directory "../../.coverage/temp/{1}" \
            ts-mocha \
            --config ../../.mocharc.cjs \
            -- \
            "**/*.test.ts" \
        ;
    ' \
    ::: \
    package1 \
    package2 \
&& echo "Passed" \
|| echo "Failed" \
;

Assuming you have a folder structure like as follows:

/.c8rc.json
/.coverage/report/
/.coverage/temp/
/.mocharc.cjs
/packages/package1/
/packages/package2/

With actual NPM-based Typescript repositories checked out in the package1 and package2 folders.

Here are the config file contents:

.c8rc.json

{
    "include": [".github/scripts/**", "packages/*/src/**", "src/**"],
    "exclude": ["**/*.test.ts", "**/*.test.tsx", "dist", "test", "**/cli.*"],

    "branches": 80,
    "functions": 80,
    "lines": 80,
    "statements": 80,

    "watermarks": {
        "branches": [80, 95],
        "functions": [80, 95],
        "lines": [80, 95],
        "statements": [80, 95]
    },

    "all": true,
    "check-coverage": true,
    "skip-empty": true,
    "skip-full": true,

    "extension": [".ts", ".tsx"]
}

.mocharc.cjs

const Path = require('node:path');
const FS = require('node:fs');

// = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
const config = {
    exit: true,
    'node-option': [
        'experimental-specifier-resolution=node',
        'inspect=0.0.0.0',
        'no-warnings',
        'trace-warnings',
        'unhandled-rejections=strict',
        'loader=ts-node/esm',
    ],
    loader: 'ts-node/esm',
    parallel: false,
    require: [],
    timeout: 60000,
};

/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
const localChaiSetupPath = 'test/setup_chai.ts';
if (FS.existsSync(localChaiSetupPath)) {
    config.require.push(localChaiSetupPath);
}

/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
const localHooksPath = 'test/hooks.ts';
if (FS.existsSync(localHooksPath)) {
    config.require.push(localHooksPath);
}

/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
module.exports = config;

Versions

• OS:
  • Docker host: macOS 13.1 on an M1 Pro processor.
  • Docker guest: Debian GNU/Linux 11 (bullseye) Linux b773fd421f78 5.10.124-linuxkit #1 SMP PREEMPT Thu Jun 30 08:18:26 UTC 2022 aarch64 GNU/Linux
• NodeJS v18.13.0
• C8 7.12.0
• Mocha 10.2.0
• ts-mocha 10.0.0

[eye1]

I was able to find a workaround myself.

By setting --jobs 8

lowering number of jobs as to how many are used by default seems to fix the issue and code coverage works fine.