🚀 Feature: Be able to add flag to some it() to prevent skipping

[ChristianMayer]

Is your feature request related to a problem or a nice-to-have?? Please describe.
I'm writing a (sort of grey box) test for a computational heavy function. This function works on my (test-)data in well defined steps. Now I want to test the intermediate data for correctness, otherwise it doesn't make sense to go to the next computation step. This check is easy in the forward direction (i.e. proof its correctness, but not possible in the backward direction; just like a hash function).

E.g. something like:

describe('Computational heavy calculation`, function(){
  let intermediateResult;

  it('should calcultate step 1', function(){
    intermediateResult = step1(42);
    assert(myHash(intermediateResult) === 1234);
  });

  it('should calcultate step 2', function(){
    intermediateResult = step2(intermediateResult);
    assert(myHash(intermediateResult) === 5678);
  });

  it('should calcultate step 3', function(){
    intermediateResult = step3(intermediateResult);
    assert(myHash(intermediateResult) === 90);
  });
});

Using a tool like the IntelliJ IDE I can now easily tell mocha just to run the second it() (on the command line it's translating to a --grep for mocha.). But this will then run only the second it() and not the first one, i.e. this test will fail as the necessary data isn't there.

Describe the solution you'd like
When I could pass the it() function a parameter that tells mocha that this must not be skipped (e.g. like it('desc', function(){...}, {skip: false})), mocha could make sure that all such flagged it() calls before the one required on the command line will run as well.

A similar approach could be to add a differently named function (e.g. step()) which is just it() but with this flag.

Describe alternatives you've considered
I could monitor whether all required steps have run, and when not, just run them myself. But this creates a very complex and clumsy testing function that by itself might introduce bugs.

I have also considered to store the full intermediate result on the disk to be able to compare it with the result, but also to be able to restart from any point by reloading the data. But as my intermediateResult is a very complex JavaScript object, including some Set() and especially functions, this is no data that can sensibly be serialized.

Additional context
Researching the possibilities or alternatives I have to achieve this functionality I've seen that many others have stumbled across this limitation. They quite often would need such a functionality for smoke testing.

[JoshuaKGoldberg]

🤔 This runs counter to my intuition for how Mocha & other test frameworks are meant to be run. The ideology I've seen has generally been that tests should be isolated and not depend on each other. Otherwise an earlier test failing would "taint" the validity of all subsequent tests. For example, in the provided code, if the intermediate result from step 1 is invalid, then it would potentially lead to invalid results in steps 2 and 3.

A common response to "tests need to rely on previously expensive data" is to move the creation and caching of that data outside of the tests if possible. This refactoring of the provided code does so:

const lazy = (creator) => {
  let value;
  return () => (value ??= creator());
};

const values = {
  // (pretending these all are computationally intense)
  step1: lazy(() => 1),
  step2: lazy(() => values.step1() ** 2),
  step3: lazy(() => values.step2() ** 3),
};

describe("Computational heavy calculation", function () {
  it("should calculate step 1", function () {
    const result = values.step1();
    assert(myHash(result) === 123);
  });

  it("should calculate step 2", function () {
    const result = values.step2();
    assert(myHash(result) === 456);
  });

  it("should calculate step 3", function () {
    const result = values.step3();
    assert(myHash(result) === 789);
  });
});

Note that this isn't always possible. It's just a theoretical other strategy that some tests are able to take.

Researching the possibilities or alternatives I have to achieve this functionality I've seen that many others have stumbled across this limitation. They quite often would need such a functionality for smoke testing.

Could you provide references, please? What are the links to others who'd want that functionality for smoke testing?

I'm not saying I'm right to push back and you're wrong to use that strategy 😄. Just that I have neither seen nor sought out any well-substantiated demand for using Mocha in this way. So if there's a major way folks want to use Mocha, we'll want to see the reasoning why to be able to accommodate it.

[ChristianMayer]

The ideology I've seen has generally been that tests should be isolated and not depend on each other. Otherwise an earlier test failing would "taint" the validity of all subsequent tests. For example, in the provided code, if the intermediate result from step 1 is invalid, then it would potentially lead to invalid results in steps 2 and 3.

This refactoring [...]

This refactoring doesn't solve the fact, that the result of step2 depends on the result of step1 and the result of step3 needs step2 and thus also step1.
So when there it a bug in step1 all the other steps will fail.

Writing the tests in such a way where each test doesn't know anything about the other tests will result in a test log with many fails - although only step1 is faulty.
When you have a step() instead of an it() (or an it(..., {skip:false}) then the test runner knows the dependancies and will not run the tests right after the failing one and clutter the log with many false positives.

Researching the possibilities or alternatives I have to achieve this functionality I've seen that many others have stumbled across this limitation. They quite often would need such a functionality for smoke testing.

Could you provide references, please?

When I wrote this issue I did some research and found others that stumbled about the same issue. There I even learned the word "smoke test" that I didn't know before :) To give a reference here I have to google again.

[JoshuaKGoldberg]

It's been a while and nobody else has posted interest here. Given #5027, we're really trying not to make big changes - and that includes big features specific to Mocha like this. I'm going to go ahead and say this is out of scope. Thanks for filing + posting details though, this was an interesting one.

If you can get this adopted by other test frameworks -the process for which would presumably include convincing lots of folks this is broadly useful- then we'd be happy to re-evaluate here. Cheers! 🤎