Fix flaky Playwright Tests

[grahamlangford]

Motivation

• Playwright runs have become increasingly flaky lately
• We're increasingly seeing Playwright failures and deciding to re-run the tests rather than investigate
  • This slows development
  • Means real failures are not investigated immediately

Acceptance Criteria

• Recent flaky tests are identified
• Solutions to prevent tests from failing are implemented
• If there are any best practices we should be using to prevent flaky tests, this is shared with the team

[grahamlangford]

@Pashaminkovsky I'm bumping the priority on this ticket because it is starting to heavily impact velocity (we're having to re-run our e2e tests 2-3 times to get them to pass, this is really harming velocity)

[grahamlangford]

Known Flaky Tests:

• tests/extensionConsole/activation.spec.ts › can activate a mod via url
• tests/extensionConsole/modsPage.spec.ts › can open mod in the workshop
• tests/modLifecycle.spec.ts:28:5 › create, run, package, and update mod
• tests/pageEditor/addStarterBrick.spec.ts:153:5 › Add starter brick to mod
• tests/pageEditor/saveMod.spec.ts:26:5 › can save a new trigger mod
• tests/runtime/sidebar/sidebarNavigation.spec.ts:25:5 › sidebar mod panels are persistent during navigation
• tests/smoke/floatingActionButton.spec.ts:51:7 › sidebar page smoke test › can hide the floating action button
• tests/smoke/modsPage.spec.ts › extension console mods page smoke test › can view available mods
• tests/telemetry/errors.spec.ts:126:5 › can report a service worker error to telemetry service

[fungairino]

Some more:

• tests/pageEditor/brickConfiguration.spec.ts:28
• tests/smoke/workshopPage.spec.ts › extension console workshop smoke test › can navigate to workshop page without a username
• tests/deployments/deploymentActivation.spec.ts › activate a deployed mod in the extension console

[fungairino]

Most of the e2e flakiness was due to latency issues which are solved temporarily by bumping up timeouts. Following up on backend performance separately.

[fungairino]

Following up on a few more flaky tests:

• authenticate with unaffiliated user
• create, run, package, and update mod
• Add new mod with different starter brick components