Feature: Stop Cypress when there are too many failed tests

[dcastil]

Current behavior:

Sometimes a feature I work on breaks a lot of tests in the CI build. I have to wait until all tests are finished until I can figure out what happened, which can take very long due to long timeouts.

Desired behavior:

Being able to set a variable in cypress.json to stop the Cypress test execution and fast-forward to uploading the data to the Cypress dashboard when a specific number of test failures is exceeded.

For example in cypress.json:

{
  "maxFailedTests": 10
}

[jennifer-shehane]

We've definitely discussed this and other variants of this internally in terms of improving 'failure workflow'. Thanks for opening this issue!

[dcastil]

Here is an example of what I mean.

I know that it would be better to put this just into one it() block, but then it would get hard to figure out at which point the test failed.

[brian-mann]

Yah, you are sort of "going against the grain" here where you are splitting up things across multiple tests but it's really just a single test, which is an anti pattern.

https://docs.cypress.io/guides/references/best-practices.html#Having-tests-rely-on-the-state-of-previous-tests

I can understand what you're doing - and regardless I think the maxFailedTests is a good option, but if you were say writing a single test or using a beforeEach you naturally would not have this problem.

You say that it's hard to tell at which point the test failed, but I'm not seeing why. Cypress records screenshots and videos, and you can use cy.log() in your tests to add context to where the test is. With these it should be immediately obvious where something did or didn't fail. We also have the Dashboard server which records everything making this process even easier.

Can you go into detail as to your process or why you're finding this challenging?

[dcastil]

Yes, you're right. I should try cy.log() instead, I didn't consider that.

My test in the screenshot is going through a registration process with several form pages. The test often fails because newly added form fields are not yet handled in Cypress and the website doesn't let Cypress press the "next"-button. In the error message Cypress provides information about the button element, but not the context of the element (e.g. the URL or previous commands), so I can't easily figure out which of the many "next"-buttons I click in the registration process caused the error.

I know that I can look it up in the screenshots within the dashboard, but I would like to keep the dashboard for more detailed investigation and make such basic things like checking where the test failed visible within the CI logs. And because Cypress only logs whether an it() block passed, I separated the test into several such blocks.

[dcastil]

@brian-mann I just tested it but Cypress doesn't log cy.log()s to the console. Is there a way to print a status from within a test? So I have something like this in the console:

log: logs in 'proposal-submitted-2' without UI
log: navigates through recap
log: selects joint account
…
✓ registration joint account happy path (26867ms)

[beaugunderson]

@dcastil the docs for cy.task include an example for logging to the console 👍

[beaugunderson]

and just to link the issues; this is very similar to the feature requested in #518

[jennifer-shehane]

Closing as duplicate of #518