bug(forge test): --fail-fast flag does not work as it is not applied across multiple test suites

[mds1]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (13af418 2023-12-05T00:21:30.568484000Z)

What command(s) is the bug in?

forge test --fail-fast

Operating System

None

Describe the bug

Running FOUNDRY_FUZZ_RUNS=100000 forge test --fail-fast on the below code should run test_Increment and testFuzz_SetNumber in parallel, and fail test_Increment should fail immediately. However, forge test does not fail and exit until all 100,000 fuzz runs complete

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

import {Test, console2} from "forge-std/Test.sol";
import {Counter} from "../src/Counter.sol";

contract CounterTest is Test {
    Counter public counter;

    function setUp() public {
        counter = new Counter();
        counter.setNumber(0);
    }

    function test_Increment() public {
        assertTrue(false);
    }

    function testFuzz_SetNumber(uint256 x) public {
        counter.setNumber(x);
        assertEq(counter.number(), x);
    }
}

[grandizzy]

looking at code it looks like --fail-fast applies for different test suites and not for tests within same suite, considering a test contract like

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;

import {Test, console2} from "forge-std/Test.sol";
import {Counter} from "../src/Counter.sol";

contract ACounterTest is Test {
    Counter public counter;

    function setUp() public {
        counter = new Counter();
        counter.setNumber(0);
    }

    function test_Increment() public {
        assertTrue(false);
    }

    function testFuzz_AnotherSetNumber(uint256 x) public {
        counter.setNumber(x);
        assertEq(counter.number(), x);
    }
}

contract BCounterTest is Test {
    Counter public counter;

    function setUp() public {
        counter = new Counter();
        counter.setNumber(0);
    }

    function testFuzz_SetNumber(uint256 x) public {
        counter.setNumber(x);
        assertEq(counter.number(), x);
    }
}

if running without flag then both contracts are tested (even though test_Increment in first contract fails)

FOUNDRY_FUZZ_RUNS=2000 forge test

Ran 2 tests for test/Counter.t.sol:ACounterTest
[PASS] testFuzz_AnotherSetNumber(uint256) (runs: 2000, μ: 27500, ~: 28366)
[FAIL. Reason: assertion failed] test_Increment() (gas: 10766)
Test result: FAILED. 1 passed; 1 failed; 0 skipped; finished in 97.51ms

Ran 1 test for test/Counter.t.sol:BCounterTest
[PASS] testFuzz_SetNumber(uint256) (runs: 2000, μ: 27630, ~: 28387)
Test result: ok. 1 passed; 0 failed; 0 skipped; finished in 99.50ms

Ran 2 test suites in 197.01ms: 2 tests passed, 1 failed, 0 skipped (3 total tests)

Failing tests:
Encountered 1 failing test in test/Counter.t.sol:ACounterTest
[FAIL. Reason: assertion failed] test_Increment() (gas: 10766)

Encountered a total of 1 failing tests, 2 tests succeeded

when run with fail fast the 2nd suite is not executed anymore

FOUNDRY_FUZZ_RUNS=2000 forge test --fail-fast

Ran 2 tests for test/Counter.t.sol:ACounterTest
[PASS] testFuzz_AnotherSetNumber(uint256) (runs: 2000, μ: 27540, ~: 28366)
[FAIL. Reason: assertion failed] test_Increment() (gas: 10766)
Test result: FAILED. 1 passed; 1 failed; 0 skipped; finished in 99.20ms

Ran 1 test suite in 99.20ms: 1 tests passed, 1 failed, 0 skipped (2 total tests)

Failing tests:
Encountered 1 failing test in test/Counter.t.sol:ACounterTest
[FAIL. Reason: assertion failed] test_Increment() (gas: 10766)

Encountered a total of 1 failing tests, 1 tests succeeded

looks like a design choice not a bug?

[grandizzy]

proposed way to interrupt runs for fuzz / invariant tests proptest-rs/proptest#460 to be applied in addition to using rayon collect / short-circuit on Err

[zerosnacks]

Just confirming this is still an active issue, setup as described still yields:

FOUNDRY_FUZZ_RUNS=1000000 forge test --fail-fast
[⠊] Compiling...
No files changed, compilation skipped

Ran 2 tests for test/Counter.t.sol:CounterTest
[PASS] testFuzz_SetNumber(uint256) (runs: 1000000, μ: 31045, ~: 31288)
[FAIL. Reason: assertion failed] test_Increment() (gas: 3071)
Suite result: FAILED. 1 passed; 1 failed; 0 skipped; finished in 22.34s (22.34s CPU time)

Ran 1 test suite in 22.34s (22.34s CPU time): 1 tests passed, 1 failed, 0 skipped (2 total tests)

Failing tests:
Encountered 1 failing test in test/Counter.t.sol:CounterTest
[FAIL. Reason: assertion failed] test_Increment() (gas: 3071)

Encountered a total of 1 failing tests, 1 tests succeeded