--timeout-multiplier 20

[ArtificialOwl]

No idea if we should do this, but at last we have some green in the checklist

[PVince81]

👍

[danxuliu]

I would not increase the timeout multiplier.

The timeouts in the acceptance tests should be enough to run them in a normal machine; the multiplier is used when running in CI to ensure that even if the system is very loaded and the browser is slow there will be enough time for the elements to appear in the DOM and things like that. However, a multiplier of 10 should be more than enough; for example, with that multiplier, the tests will wait up to 20 seconds in the shortest waits.

Moreover, the flaky tests are likely not caused by the tests giving up on finding the elements too fast; they are probably caused by other kind of interactions. Back in the day a hack was needed to reopen the sharing menu in some cases; looking at the code now I think that the problem could be that, sometimes, the menu is seen as closed before the click to open it had time to actually open it, which causes a new click which, combined with the previous one, in the end cause the menu to be closed 🤔

[come-nc]

Is there anyway we could get a screenshot from acceptance tests on failure? This would help a lot debugging and detecting false negative from real bugs.

[danxuliu]

Is there anyway we could get a screenshot from acceptance tests on failure? This would help a lot debugging and detecting false negative from real bugs.

I have been wanting to add that for ages, but unfortunately there is no such feature right now.

My main question would be, where to store the screenshots or videos when the acceptance tests fail in CI? Do we have any storage where they could be uploaded (even if they get wipped out after some days)?

[come-nc]

My main question would be, where to store the screenshots or videos when the acceptance tests fail in CI? Do we have any storage where they could be uploaded (even if they get wipped out after some days)?

Can I be as bold as suggesting we store those on Nextcloud? 🤓

[ArtificialOwl]

I have been wanting to add that for ages, but unfortunately there is no such feature right now.

what is the idea here ? store the whole video and keep timestamp somewhere or only the last few minutes before the issue ?

[danxuliu]

I have been wanting to add that for ages, but unfortunately there is no such feature right now.

what is the idea here ? store the whole video and keep timestamp somewhere or only the last few minutes before the issue ?

Each test scenario is fully isolated, so the idea would be to start a recording when the scenario starts and stop it when the scenario ends. If the test failed the video is uploaded somewhere and a link is added to the CI output, and if not the video is discarded (of course in some cases it would be good to have a video even if it worked in case it was a false positive, but... for the common CI case we could ignore that and leave that as an option when locally run).

Alternatively it would be possible to take screenshots on each step, but that would not tell the full history. On the other hand, screenshots may help to check the UI state when something failed, as in the case of a video the browser might shut down too fast to see anything useful. So maybe the best would be to have both.