Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Improve tests #4372

Merged
merged 12 commits into from
Apr 3, 2023
Merged

chore: Improve tests #4372

merged 12 commits into from
Apr 3, 2023

Conversation

JorTurFer
Copy link
Member

@JorTurFer JorTurFer commented Mar 14, 2023 •
edited
Loading

Changes done by this PR:

  • K8s v1.26 is used for functional tests
  • main e2e test are executed in an ARM64 runner as they are faster
  • WaitForNamespaceDeletion (the helper we use in e2e test to wait until namespace is deleted) timeout has increased from 3 minutes to 10 minutes
  • run-all.sh has been removed and another run-all.go script has been added to improve the test execution maintaining always X test on going instead of batches of X, having to wait until the whole batch finishes for starting the next batch
  • create_cas_volume.yml has been removed in favor of patch_operator.yml which also sets the requested cpu to 1 for operator during e2e tests
  • Flaky test in controllers/keda/scaledobject_controller_test.go has been fixed
  • polling_cooldown_test has been updated because it was overloading the CPU due to a misconfiguration and duplicated test cases have been removed
  • cassandra_test has been refactored as it was a flaky test (now it uses helm chart for cassandra and jobs)
  • couchdb_test has been refactored as it was a flaky test
  • etcd_cluster_test has been refactored as it was a flaky test
  • mongodb_test has been refactored as it was a flaky test (now it uses helm chart for mongodb)
  • nats_cluster_test has been refactored to reduce errors in logs
  • nats_helper has been update to retry jobs on failures
  • redis_helper has been updated to use helm chart to deploy a redis cluster
  • chaos_test has been renamed as disruption_test and moved to sequential execution folder
  • prometheus_metrics_test has been moved to sequential execution folder to avoid interactions from other tests
  • setup_test has been refactored to optimize the spin up process deploying all the webhooks and waiting all of them at once instead of waiting them one by one before continuing

And which is the result? (ignore the building time as those steps do different things in each case)

main e2e test needs > 2 hours
image

e2e test on pr needs < 1 hour
image
image

I can expect that the needed time on main e2e tests will be similar to current times on PRs

Checklist

  • Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #2829
Fixes #4244

@JorTurFer JorTurFer force-pushed the improve-e2e branch 2 times, most recently from a79f156 to 64c7fad Compare March 27, 2023 21:28
@JorTurFer JorTurFer changed the title WIP - chore: Increase concurrent e2e tests WIP - chore: Improve tests Mar 27, 2023
@JorTurFer JorTurFer force-pushed the improve-e2e branch 8 times, most recently from 69aafaf to ddbc899 Compare March 30, 2023 21:39
@JorTurFer JorTurFer changed the title WIP - chore: Improve tests chore: Improve tests Mar 31, 2023
@JorTurFer JorTurFer marked this pull request as ready for review March 31, 2023 12:04
@JorTurFer JorTurFer requested a review from a team as a code owner March 31, 2023 12:04
@JorTurFer
Copy link
Member Author

JorTurFer commented Mar 31, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@zroubalik
Copy link
Member

/run-e2e Update: You can check the progress here

Failed tests:
Execution of tests/internals/polling_cooldown_so/polling_cooldown_so_test.go, has failed after 3 tries

@JorTurFer
Copy link
Member Author

JorTurFer commented Mar 31, 2023 •
edited
Loading

Yeah, I have seen it.
IDK how many times I have triggered them without fails, but I still prefer to detect them here and not after merging the changes 😄

Let's start again

@JorTurFer
chore: Improve tests
d0d3a70 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer JorTurFer force-pushed the improve-e2e branch 2 times, most recently from 9181edf to 7c2f36b Compare March 31, 2023 16:28
@JorTurFer
update polling_cooldown to fix flaky scenarios
d0a5d49 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
Copy link
Member Author

JorTurFer commented Mar 31, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Mar 31, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
use k8s 1.26 assets
d35a2d3 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
fix flacky tests
bbea2a2 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
Copy link
Member Author

Let's start again

@JorTurFer
fix styles
9676abc 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
update cassandra test to use helm chart
044ea2b 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
update disruption_test
0bcb7c8 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
use mongo chart for mongo tests
a7218b5 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
fix styles
0ccb57a 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
fix cassandra using jobs for actions
2778bdf 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
wait and remove jobs for polling-colldown tests
25d095d 
Signed-off-by: Jorge Turrado <jorge_turrado@hotmail.es>
@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 2, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 3, 2023 •
edited
Loading

/run-e2e Update: You can check the progress here

This has failed because the namespace deletions have failed. IDK if there is something that I can do to mitigate this or if it has been a transient fail:
image

Maybe increasing the deletion timeout?

EDIT: I have increased the timeout from 3 minutes to 10

@JorTurFer
update namespace deletion timeout from 30 seconds to 2 minutes
1cfd786 
Signed-off-by: Jorge Turrado <jorge.turrado@scrm.lidl>
@JorTurFer
Copy link
Member Author

JorTurFer commented Apr 3, 2023 •
edited by github-actions bot
Loading

/run-e2e
Update: You can check the progress here

zroubalik
zroubalik approved these changes Apr 3, 2023
View reviewed changes
Copy link
Member

@zroubalik zroubalik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tremendous work!

@zroubalik zroubalik merged commit e979077 into kedacore:main Apr 3, 2023
@JorTurFer JorTurFer deleted the improve-e2e branch April 3, 2023 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zroubalik zroubalik zroubalik approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create sequential or standalone package in e2e test suite Functional tests are unstable
2 participants
@JorTurFer @zroubalik