Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up spaces tests by letting v2 migrations do less work #91829

Merged
merged 5 commits into from
Feb 22, 2021

Conversation

rudolf
Copy link
Contributor

@rudolf rudolf commented Feb 18, 2021

Summary

When I added v2 migrations to FTR (#89992) I kept all the esArchiver datasets as-is so that we're sure the behaviour is exactly the same between v1 and v2.

However, given the discussion in #91618 we are looking for ways to optimize CI runtime. One way to achieve better performance is to change esArchives so that the v2 migration algorithm has to do less work. Instead of a migration from a legacy or v1 index, the v2 migration will now see a v2 migrations index and will only update mappings and transform documents (skipping three expensive operations create index, clone, reindex).

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@rudolf rudolf added the project:ResilientSavedObjectMigrations Reduce Kibana upgrade failures by making saved object migrations more resilient label Feb 18, 2021
@rudolf
Copy link
Contributor Author

rudolf commented Feb 19, 2021

Brings CI down to 1h57 🎉

@rudolf rudolf marked this pull request as ready for review February 19, 2021 11:04
@rudolf rudolf requested review from a team as code owners February 19, 2021 11:04
@rudolf rudolf added v7.12.0 v8.0.0 Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc labels Feb 19, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@rudolf rudolf added the release_note:skip Skip the PR/issue when compiling release notes label Feb 19, 2021
Copy link
Contributor

@jportner jportner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, what a simple change!

Brings CI down to 1h57 🎉

What's the delta? 😄
Edit: In the linked issue, Spencer said "the overall build time changed from ~1h45m to ~2h10m". So I'm assuming 1h57m is down from 2h10m. Pretty good!

QQ: I noticed that x-pack/test/saved_object_api_integration is not listed in CODEOWNERS. Do you want to add @elastic/kibana-security as the code owner for that?

@rudolf rudolf enabled auto-merge (squash) February 22, 2021 11:50
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@rudolf rudolf merged commit 42b814e into elastic:master Feb 22, 2021
@rudolf rudolf deleted the faster-v2-migrations branch February 22, 2021 13:47
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 24, 2021
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

4 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

5 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

2 similar comments
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

@spalger
Copy link
Contributor

spalger commented Mar 15, 2021

@rudolf I'm sorry, I know we've talked about this but what's the status of this backport?

@rudolf
Copy link
Contributor Author

rudolf commented Mar 15, 2021

I was blocked on #94420 but can now backport this to 7.x

@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

rudolf added a commit to rudolf/kibana that referenced this pull request Mar 17, 2021
…91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json
rudolf added a commit to rudolf/kibana that referenced this pull request Mar 17, 2021
…91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

1 similar comment
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

rudolf added a commit that referenced this pull request Mar 18, 2021
…1829) (#94797)

* Speed up spaces tests by letting v2 migrations do less work (#91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json

* Apply suggestions from code review

* Fix backport for ES _types

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 19, 2021
rudolf added a commit that referenced this pull request Mar 19, 2021
…91829) (#94801)

* Speed up spaces tests by letting v2 migrations do less work (#91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json

* Fix backport for ES _types

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
project:ResilientSavedObjectMigrations Reduce Kibana upgrade failures by making saved object migrations more resilient release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.12.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants