Test passes when loading a page which returns 500

[trygveaa]

Current behavior:

If a page which is navigated to during a test returns HTTP status 500, the test still passes.

Note that the test does fail when visiting the page directly with .visit, it only passes when loaded by normal navigation, e.g. by clicking a link or submitting a form.

Desired behavior:

I think the test should fail when a HTTP error is encountered on page load.

Steps to reproduce:

This test reproduces the issue:

describe('500 test', () => {
    it('should fail', () => {
        cy.visit('/');
        cy.get('[href="/500"]').click();
    });
});

With this config:

{
    "baseUrl": "https://httpstat.us/",
    "chromeWebSecurity": false
}

Note that I had to disable web security for this particular page, otherwise I get a cross origin error. Since it is a relative link, that shouldn't be necessary. But if I test with a my own page, I don't get a cross origin error, so this seems to be a separate issue.

Versions

Cypress 3.1.2 using the embedded Electron 59 on Arch Linux.

[bahmutov]

yeah, I tried the above code, does, in fact, redirect to the page that has content, but status is 500

when clicking on the underlined 500 in the stack trace goes to a blank source page, which is also weird by itself

[jennifer-shehane]

Duplicate of #2825 #910

The issue already opened proposes an API to allow defined statuses that should fail any test where they occur.

Edit: Fix wrong issue link

[trygveaa]

@jennifer-shehane: Which issue? You referenced this issue.

The visit command already fails for some statuses, so I think it would be natural that any page load with the same statuses makes the test fail, not just loads from the visit command. So I don't think you necessarily need an API to define statuses to solve this.

[brian-mann]

A 500 is a valid status code. When you're acting like a user clicking links and stuff, then you're effectively in the e2e layer which is all about the user experience and is whatever the server serves. In this way, you're limited to testing things like the URL and the DOM directly.

What I mean is - I don't think there's any guarantee that when you're testing that the test should necessarily fail when any request comes back 500. We've discussed exposing some API's that upon page navigation you can access the headers, status code, body, etc programmatically and then enable you to test it directly.

However today, if you want to test the route programmatically, I suggest using cy.request() since you don't care about the actual HTML results.

cy.get('href="/500"').invoke('prop', 'href').then((href) => {
  // now makes a programmatic request to this HREF and will fail if it returns 500
  cy.request(href) 
})

[bahmutov]

I get that it is valid browser behavior - just continue on 500 when visiting a page, and it’s a big if. But we could be much more helpful here and at least show a warning pop up and output to the console - that you just loaded a page with 500 status

…

Sent from my iPhone

On Nov 26, 2018, at 15:09, Brian Mann ***@***.***> wrote: A 500 is a valid status code. When you're acting like a user clicking links and stuff, then you're effectively in the e2e layer which is all about the user experience and is whatever the server serves. In this way, you're limited to testing things like the URL and the DOM directly. However if you want to test the route programmatically, I suggest using cy.request() since you don't care about the actual HTML results. cy.get('href="/500"').invoke('prop', 'href').then((href) => { cy.request(href) // now makes a programmatic request to this HREF and will fail if it returns 500 }) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[trygveaa]

A 500 is a valid status code. When you're acting like a user clicking links and stuff, then you're effectively in the e2e layer which is all about the user experience and is whatever the server serves. In this way, you're limited to testing things like the URL and the DOM directly.

Well, 500 is used when the server encountered an unexpected error, which I don't think can/should be any valid scenario. But I see your point for 4xx codes, which might be a valid scenario that you might want to test without failure.

What I mean is - I don't think there's any guarantee that when you're testing that the test should necessarily fail when any request comes back 500.

Yeah, I agree that it might not be practical for all requests. What I was thinking about was just failing for page loads, to make them consistent with the visit command. But regarding the above, maybe consistency with visit isn't a good idea after all.

However today, if you want to test the route programmatically, I suggest using cy.request() since you don't care about the actual HTML results.

I don't want to test the route programmatically. My use case for this request is that the test fails if the backend encounters an uncaught exception.

[jennifer-shehane]

Sorry about the wrong link. I meant this is a duplicate of #910

[trygveaa]

That's for XHR's though. I'm talking about page loads. But maybe you want to handle them the same way?