Add a Cypress Test 🌲 #8295

dbkr · 2022-04-12T14:01:32Z

A Cypress clone of the 'register an account' part of the end-to-end tester.

Queues Cypress tests for after the layered build is done
Downloads the layered build & fires up a server
Runs Cypress, testing that the process of registering an account with a terms-of-service prompt to accept works
Includes a Cypress plugin to start & stop instances of synapse on demand in docker containers
Uploads screenshots, videos & docker logs as an artifact on failure

Note that the trickiest end-to-end test to replicate will be the actual sending an encrypted message from one client to another, since Cypress doesn't support running two browsers/tabs at once. I'd currently say the best way to do this would be to test sending and receiving separately with a bot on the other end. Porting this over doesn't need to block us from writing other Cypress tests though.

Another open problem is how to factor tests like this into our coverage.

This change is marked as an internal change (Task), so will not be included in the changelog.

Preview: https://pr8295--matrix-react-sdk.netlify.app
⚠️ Do you trust the author of this PR? Maybe this build will steal your keys or give you malware. Exercise caution. Use test accounts.

Plus cypress plugins to manage synapses in docker containers

codecov · 2022-04-12T14:08:06Z

Codecov Report

Merging #8295 (dd0ff8e) into develop (ecdc11d) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop    #8295   +/-   ##
========================================
  Coverage    29.79%   29.79%           
========================================
  Files          872      872           
  Lines        50006    50006           
  Branches     12723    12723           
========================================
  Hits         14897    14897           
  Misses       35109    35109

also actually give the test a name

also document the chrome setting

and remove debug logging / comment fixes

and temporarily remove contional to test

to be compatible with fs-extra types

also make password more... sensible

Seems to be unnecessary since the signing key is perfectly fine

turt2live

otherwise lgtm - many of these are questions, but also Changes Requested

turt2live · 2022-04-12T23:26:45Z

.github/workflows/layered-build.yaml

@@ -20,4 +20,29 @@ jobs:
                  path: element-web/webapp
                  # We'll only use this in a triggered job, then we're done with it
                  retention-days: 1
-
+    cypress:


not sure this is in the right workflow file...? "layered build" doesn't scream tests, at least.

ah yes, unfortunately I think it can only run after a job in the same workflow file (apart from the workflow_run job but that runs with too much permission). Meant to rename the workflow file, and have now done so.

cypress/integration/1-register/register.spec.ts

turt2live · 2022-04-12T23:28:23Z

cypress/plugins/synapsedocker/index.ts

+function randB64Bytes(numBytes: number): string {
+    return crypto.randomBytes(numBytes).toString("base64").replace(/=*$/, "");
+}


we should already have a random string utility we can use?

Mmm, true - I guess importing code from the app would be fine, but it feels like it's probably good hygiene to keep them completely separate, especially when the code in question is as simple as it is. Plus this is node-specific and we specifically want base64 encoding for the ed25519 key.

cypress/plugins/synapsedocker/index.ts

turt2live · 2022-04-12T23:29:36Z

cypress/plugins/synapsedocker/index.ts

+    const tempDir = await fse.mkdtemp(path.join(os.tmpdir(), 'react-sdk-synapsedocker-'));
+
+    // change permissions on the temp directory so the docker container can see its contents
+    await fse.chmod(tempDir, 0o777);


20% sure this won't work on windows, but also a good chance that nodejs will have stubbed it

Nodejs says, "on Windows only the write permission can be changed, and the distinction among the permissions of group, owner or others is not implemented." so sounds like it will work to some extent, or at least no-op.

turt2live · 2022-04-12T23:30:59Z

cypress/plugins/synapsedocker/index.ts

+                    "-d",
+                    "-v", `${synCfg.configDir}:/data`,
+                    "-p", "8008/tcp",
+                    "matrixdotorg/synapse",


we've previously used develop to ensure forwards compatibility with upcoming features, though there's certainly an argument to be had about whether that's a good idea.

ah yep - let's keep it as-is for now while we have the argument

turt2live · 2022-04-12T23:31:50Z

cypress/plugins/synapsedocker/index.ts

+            console.log(`Starting synapse with config dir ${synCfg.configDir}...`);
+
+            const synapseId = await new Promise<string>((resolve, reject) => {
+                childProcess.execFile('docker', [


can we also give this a --name and set --rm so it cleans itself up upon exiting?

Have given them a name. We collect logs at the end so if we use --rm then we don't have a chance to get the logs out. I have also made it tidy up any remaining synapses after each spec run to try to avoid containers sitting around if people forget to stop them in the tests.

cypress/plugins/synapsedocker/index.ts

Co-authored-by: Travis Ralston <travisr@matrix.org>

Also: * Move the synapseStart / synapseStop functions out to the top level so they can be reused * Add a tsconfig file * Give the containers names

We don't upload it anyway so tell cypress not to so it can not bother encoding them

and fix existing lint errors

and make it pass the type checks, specifically: * Upgrade sinon fake timers to a version that has the right types * Set module resolution * Type check cypress files separately

Probably better to just call it an element web build

turt2live · 2022-04-13T19:07:22Z

cypress/plugins/synapsedocker/index.ts

+    await new Promise<void>((resolve, reject) => {
+        childProcess.execFile('docker', [
+            "rm",
+            id,
+        ], err => {
+            if (err) reject(err);
+            resolve();
+        });
+    });


might be best to put this on a try { } finally { ... }, but not blocking for this PR

* A first, maybe working cypress test Plus cypress plugins to manage synapses in docker containers * Fix yaml * This file is important * try & find where it's put the artifact * Download artifact to a directory * pics or it didn't happen * Add conditional, otherwise no artifacts on failure... * Try increasing timeout also actually give the test a name * Try in chrome * Get docker logs to see why it's failing also document the chrome setting * Try changing mode on homeserver.yaml * debug * More debugging * more file permissions debugging * ARGH * more debug * sigh * Eugh, that's not how arguments work * Add the option to really allow open registration and remove debug logging / comment fixes * failure to yaml * Upload docker logs as artifacts and temporarily remove contional to test * Put the conditional back * Upgrade types in end to end tests to be compatible with fs-extra types * Try reducing timeout a bit also make password more... sensible * Hex is not octal * Remove file mode Seems to be unnecessary since the signing key is perfectly fine * Give the log files extensions * Rename workflow file now it also does tests * Add cypress scripts * copyright headers * Use ? operator Co-authored-by: Travis Ralston <travisr@matrix.org> * Use develop synapse image * Tidy up any remaining synapses after each spec run Also: * Move the synapseStart / synapseStop functions out to the top level so they can be reused * Add a tsconfig file * Give the containers names * Don't upload video on test pass We don't upload it anyway so tell cypress not to so it can not bother encoding them * Enable linting on cypress files and fix existing lint errors * Type check cypress files and make it pass the type checks, specifically: * Upgrade sinon fake timers to a version that has the right types * Set module resolution * Type check cypress files separately * Rename workflow file again Probably better to just call it an element web build * Don't plus + characters in container name * Fix yaml * Stream logs to file * Add note to end to end tester to sya what's been ported * Put docker rm in finally block Co-authored-by: Travis Ralston <travisr@matrix.org>

MadLittleMods · 2022-04-19T01:35:34Z

cypress/integration/1-register/register.spec.ts

+        cy.task<SynapseInstance>("synapseStart", "consent").then(result => {
+            synapseId = result.synapseId;
+            synapsePort = result.port;
+        });


Anti-Pattern: Trying to start a web server from within Cypress scripts with cy.exec() or cy.task() .

-- https://docs.cypress.io/guides/references/best-practices#Web-Servers

I assume we've weighed the risks here?

A first, maybe working cypress test

8d22be3

Plus cypress plugins to manage synapses in docker containers

dbkr added the T-Task Refactoring, enabling or disabling functionality, other engineering tasks label Apr 12, 2022

Merge remote-tracking branch 'origin/develop' into dbkr/cypress

0854ee0

dbkr added 24 commits April 12, 2022 15:11

Fix yaml

6308954

This file is important

f4f7cac

Merge remote-tracking branch 'origin/develop' into dbkr/cypress

53a30ed

try & find where it's put the artifact

8312ead

Download artifact to a directory

e2be179

pics or it didn't happen

2c6265d

Add conditional, otherwise no artifacts on failure...

0ad99d7

Try increasing timeout

358b692

also actually give the test a name

Try in chrome

7ab9222

Get docker logs to see why it's failing

67a59bd

also document the chrome setting

Try changing mode on homeserver.yaml

2d8200b

debug

b2adcf5

More debugging

d9861a8

more file permissions debugging

d08597d

ARGH

322a750

more debug

02965aa

sigh

6c0acdb

Eugh, that's not how arguments work

20a6963

Add the option to really allow open registration

9f6128c

and remove debug logging / comment fixes

failure to yaml

41030ec

Upload docker logs as artifacts

b7c7f64

and temporarily remove contional to test

Put the conditional back

e2b7235

Merge remote-tracking branch 'origin/develop' into dbkr/cypress

0ef494a

Upgrade types in end to end tests

bf80b65

to be compatible with fs-extra types

dbkr changed the title ~~Cypress~~ Cypress 🌲 Apr 12, 2022

dbkr changed the title ~~Cypress 🌲~~ Add a Cypress Test 🌲 Apr 12, 2022

dbkr marked this pull request as ready for review April 12, 2022 20:25

dbkr requested a review from a team as a code owner April 12, 2022 20:25

dbkr added 4 commits April 12, 2022 21:29

Try reducing timeout a bit

979b5ff

also make password more... sensible

Hex is not octal

ea6f8af

Remove file mode

2aca928

Seems to be unnecessary since the signing key is perfectly fine

Give the log files extensions

38bbb55

turt2live suggested changes Apr 12, 2022

View reviewed changes

dbkr and others added 12 commits April 13, 2022 09:32

Rename workflow file now it also does tests

a1e9350

Add cypress scripts

1e8953a

copyright headers

53b8a1c

Use ? operator

1619330

Co-authored-by: Travis Ralston <travisr@matrix.org>

Use develop synapse image

b56da36

Tidy up any remaining synapses after each spec run

a78d212

Also: * Move the synapseStart / synapseStop functions out to the top level so they can be reused * Add a tsconfig file * Give the containers names

Don't upload video on test pass

4dd0545

We don't upload it anyway so tell cypress not to so it can not bother encoding them

Enable linting on cypress files

ca86bad

and fix existing lint errors

Type check cypress files

cfdb2af

and make it pass the type checks, specifically: * Upgrade sinon fake timers to a version that has the right types * Set module resolution * Type check cypress files separately

Rename workflow file again

f82aa9a

Probably better to just call it an element web build

Don't plus + characters in container name

d0cbae9

Fix yaml

13ee082

dbkr requested a review from turt2live April 13, 2022 12:03

dbkr added 2 commits April 13, 2022 15:04

Stream logs to file

58f32b7

Add note to end to end tester to sya what's been ported

d5d9aa7

turt2live approved these changes Apr 13, 2022

View reviewed changes

Put docker rm in finally block

dd0ff8e

dbkr merged commit b8013fc into develop Apr 14, 2022

dbkr deleted the dbkr/cypress branch April 14, 2022 09:42

MadLittleMods reviewed Apr 19, 2022

View reviewed changes

This was referenced Apr 19, 2022

Ask to refresh timeline when historical messages are imported (MSC2716) #8303

Closed

Ask to refresh timeline when historical messages are imported (MSC2716) #8354

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a Cypress Test 🌲 #8295

Add a Cypress Test 🌲 #8295

dbkr commented Apr 12, 2022 •

edited by github-actions bot

Loading

codecov bot commented Apr 12, 2022 •

edited

Loading

turt2live left a comment

turt2live Apr 12, 2022

dbkr Apr 13, 2022

turt2live Apr 12, 2022

dbkr Apr 13, 2022

turt2live Apr 12, 2022

dbkr Apr 13, 2022

turt2live Apr 12, 2022

dbkr Apr 13, 2022

turt2live Apr 12, 2022

dbkr Apr 13, 2022

turt2live Apr 13, 2022

dbkr Apr 14, 2022

MadLittleMods Apr 19, 2022

Add a Cypress Test 🌲 #8295

Add a Cypress Test 🌲 #8295

Conversation

dbkr commented Apr 12, 2022 • edited by github-actions bot Loading

codecov bot commented Apr 12, 2022 • edited Loading

Codecov Report

turt2live left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dbkr commented Apr 12, 2022 •

edited by github-actions bot

Loading

codecov bot commented Apr 12, 2022 •

edited

Loading