Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky core_app integration tests #92030

Merged
merged 6 commits into from
Feb 22, 2021

Conversation

afharo
Copy link
Member

@afharo afharo commented Feb 19, 2021

Summary

  • Starting off by unskipping them

Resolves #81072

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@afharo afharo added release_note:skip Skip the PR/issue when compiling release notes v7.13.0 labels Feb 19, 2021
@afharo afharo requested a review from a team as a code owner February 19, 2021 15:49
@afharo
Copy link
Member Author

afharo commented Feb 19, 2021

@elasticmachine merge upstream

Second run to see if it fails. After 3 successful runs, I'll merge.

@Bamieh
Copy link
Member

Bamieh commented Feb 22, 2021

@elasticmachine merge upstream

Copy link
Member

@Bamieh Bamieh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving to unblock in case it gets 3 successive passes.

@pgayvallet
Copy link
Contributor

Second run to see if it fails. After 3 successful runs, I'll merge.

Imho 3 green runs is far from being good enough to unskip a flaky suite (I mean, it's flaky, not red, 3 runs will likely succeed if the flakiness rate is < 10%, which is most likely the case here)

I usually fork the branch, edit the CI group tests to have only this specific test running, then use the flaky runner on 200+ iterations.

EDIT: this is not a FTR test. Well in that case, I would edit the IT file to have the test run 200 times, and then revert the commit if/when green.

@afharo
Copy link
Member Author

afharo commented Feb 22, 2021

Good call @pgayvallet! I've added a loop to run the suite 200 times. It forced me to move some of the vars and setup steps inside the describe context. Probably that was the cause of the flaky tests.

@afharo
Copy link
Member Author

afharo commented Feb 22, 2021

The tests passed 200 times in a row after the fixes mentioned in my previous comment: search src/core/server/core_app/integration_tests/default_route_provider_config.test.ts in the logs of the CI job to see them.

I'll go ahead and revert the 200 loop.

@afharo afharo enabled auto-merge (squash) February 22, 2021 12:40
@afharo afharo added the auto-backport Deprecated - use backport:version if exact versions are needed label Feb 22, 2021
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@afharo afharo merged commit 678f848 into elastic:master Feb 22, 2021
@afharo afharo deleted the unskip_core_app_tests branch February 22, 2021 14:52
kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Feb 22, 2021
* Unskip core_app integration tests

* Run the integration test 200 times

* Revert 200 execution loop

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine
Copy link
Contributor

💚 Backport successful

7.x / #92190

Successful backport PRs will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Feb 22, 2021
* Unskip core_app integration tests

* Run the integration test 200 times

* Revert 200 execution loop

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Deprecated - use backport:version if exact versions are needed release_note:skip Skip the PR/issue when compiling release notes v7.13.0
Projects
None yet
4 participants