chore: reduce windows coverage

[Rich-Harris]

We've observed that when Windows-specific test failures happen (other than those caused by vitejs/vite#9986), it's either specific unit tests or all end-to-end tests. I can't recall a time when a Windows bug has caused an end-to-end test to fail by itself.

A corollary is that we don't need to run the entire test suite in Windows, since doing so is slow as balls (#8630 notwithstanding). We can just run the unit tests and a subset of prerendering/e2e tests. That's the theory anyway.

[changeset-bot]

⚠️ No Changeset found

Latest commit: ad35f20

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[Rich-Harris]

15 minutes, still! Why is Windows so terrible?

[benmccann]

I did a review of recent PRs where there was some Windows involvement, which has made me more comfortable with this change. We should make sure that we're running the tests in packages/kit/test/build-errors as those have been highly Windows-related. As long as we're running those tests and the unit tests, I don't think we would have missed anything with this change. The one exception was #7639 which probably would have affected only a single e2e test if it were tested - however, no test exists for that one.

---

For reference, here are my notes:
#7639 - untested, but if it were tested it would likely be with an e2e test
#7507 - unit tests
#7494 - all e2e tests failing
#7059 - unit test
99fb85d - unit test
73d1d04 - unit test
#6913 & #6843 - untested, but if they were tested it would likely be with a unit test
#6796 - all e2e tests failing
fa5c424 - would likely affect multiple e2e tests
#6034 - unit test
#5739 (comment) - too difficult to test
#5265 - all e2e tests
#4997 - all e2e tests
#4923 - packaging tests
#4120 - unit tests

[benmccann]

15 minutes, still! Why is Windows so terrible?

Well I guess it would have been 30 without this change, so it's probably still a good change. I closed #8630 since I didn't see a difference with it, but folks were reporting that window-latest was only sometimes slower, but maybe it would help and I was just comparing it to a fast run?

[Rich-Harris]

Would it be worth separating e2e vs everything else into separate runs, so they can happen concurrently? Or would that just add more complexity for little gain?

[benmccann]

From that last run, the unit tests took 6s and the browser tests 10m, so I don't know there's much win in separating them

Unit tests:

Duration: 6620.07ms

Integration tests:

202 passed (10m)

There were two tests that flaked a total of three times with 45s timeouts, so we might be able to make things faster by addressing some of that flakiness:

@sveltejs/kit:test:cross-browser: 2 flaky
@sveltejs/kit:test:cross-browser: [firefox-build] › test\cross-browser\test.js:905:2 › Routing › Relative paths are relative to the current URL
@sveltejs/kit:test:cross-browser: [firefox-build-no-js] › test\cross-browser\test.js:704:2 › Routing › resets the active element after navigation

[benmccann]

Hmm. Actually I'm wondering if it ran the browser tests twice. Take a look at these log messages:

2023-01-20T17:03:28.0580491Z @sveltejs/kit:test:cross-browser: > pnpm test:unit && pnpm -r --workspace-concurrency 1 --filter="./test/**" test:cross-browser
2023-01-20T17:07:38.5301821Z @sveltejs/kit:test:cross-browser: �[32m 211 passed�[39m�[2m (4m)�[22m
2023-01-20T17:07:39.6938337Z @sveltejs/kit:test:cross-browser: > node test/setup.js && rimraf test/errors.json && playwright test test/cross-browser/
2023-01-20T17:17:12.3227011Z @sveltejs/kit:test:cross-browser: �[32m 202 passed�[39m�[2m (10m)�[22m

[Rich-Harris]

Once in dev mode (the 211 tests), once in prod (the 202 tests)