Using Middleware in Page Router causes 200 responses on prefetch or Next/Link for missing pages, and missing GetStaticProps on page nav

[broksonic21]

Link to the code that reproduces this issue

https://github.com/broksonic21/next-404

To Reproduce

In Dev mode:

• Start npm run dev
• Go to http://localhost:3000/page1
• Click Link to Missing Page
• notice it errors vs go to 404 page - because it ignored GetStaticPaths, but called the page load without GetStaticProps being called
• notice it gets a 200 for the json load for a page that's not authorized in GetStaticPaths in fallback:False mode with an empty {} response.

• Hard-refresh http://localhost:3000/missing-page - notice it works and shows 404 page.

In prod:

• Start npm run build;npm run start
• Go to http://localhost:3000/page1
• Hover over Link to Missing Page - see the prefetch returns a 200 response with a {} response body

• click the link - notice you get an application error (again, because a 200 is returned for a missing page even though it's not in GetStaticPaths in fallback:False mode, and no GetStaticProps is called)
• Hard-refresh http://localhost:3000/missing-page - notice it works and shows 404 page.

To see how it should work: for either dev or build/start:

• delete middleware.ts (which shouldn't even run - the matcher doesn't apply to these pages
• run server
• click on Link to Missing Page on http://localhost:3000/page1 - get a 404 page right away with no errors

OR

• with middleware.ts file file there
• revert to next v13.14.12
• run steps again
• see you get a 404 page, and a 404 prefetch lookup as expected

Note: issue started here: v13.4.12...v13.4.13-canary.0 - it doesn't repro in v13.4.12, and does repro in v13.4.13-canary.0 and beyond, up until current canary

Current vs. Expected behavior

Links to missing pages should go to a 404 with or without a middleware using page router, especially if the middleware doesn't match the route anyway.

See reproduction for details

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
Binaries:
  Node: 18.18.0
  npm: 10.1.0
  Yarn: 1.22.19
  pnpm: 8.8.0
Relevant Packages:
  next: 13.5.4-canary.7
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.1.3
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Middleware / Edge (API routes, runtime), Routing (next/router, next/navigation, next/link)

Additional context

I have tested against all released versions since v13.14.12, including latest canary - the issue started in v13.4.13-canary.0 and continues until today.

[broksonic21]

There's a large series of items related to middleware.ts file simply existing and pages router - anyway to get eyes on it?

• [Pages Router] - Middleware breaks URL masking #55208
• Redirects not working correctly together with middleware #56154
• Middleware cause prefetch for dynamic pages fail when changing locale #54981
• Pages render without static props on client-side navigation with skipMiddlewareUrlNormalize option #54330
• Rewrites don't work if a middleware.ts file is present #51923
• Next.js issue with middleware/catch-all/rewrites/getInitialProps when client side navigation occurs #50212

and more...

[broksonic21]

Any way to get this looked at? This has been a major hold up on a few of our initiatives and is clearly broken in pages router when using any middleware.

[broksonic21]

@ijjk normally I don't tag maintainers, so sorry for doing so, but I've seen a lot of similar issues to this one in the issue list, and this is pretty clearly broken/reproable (including in latest canary) and is currently blocking a project we are on. Any chance we can get some eyes on it?

I've collated a list of (at least these) related tickets here: #50212 (comment)

[ben-propflo]

Referencing the issue I raised here as it seems to be related to the same code change from @ijjk : #57207