Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ingest Manager] How to reduce the overhead of Kibana in the work our routes does #67989

Closed
ph opened this issue Jun 2, 2020 · 7 comments
Closed

[Ingest Manager] How to reduce the overhead of Kibana in the work our routes does #67989

ph opened this issue Jun 2, 2020 · 7 comments
Assignees
@jfsiii
Labels
discuss Team:Fleet Team label for Observability Data Collection Fleet team

Comments

@ph
Copy link
Contributor

ph commented Jun 2, 2020 •
edited by jfsiii
Loading

Our initial work showed significant CPU time

Sources of CPU time

  • Our initial research showed significant time in Saved Object related code
  • TBD

Sources of wall clock time

  • TBD
@ph ph added Team:Fleet Team label for Observability Data Collection Fleet team Ingest Management:beta1 labels Jun 2, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

@ph
Copy link
Contributor Author

ph commented Jun 2, 2020 •
edited
Loading

@roncohen / @jfsiii Can you fill it up based on your observation on the private issue?

@ph ph added the discuss label Jun 2, 2020
@jfsiii
Copy link
Contributor

jfsiii commented Jun 4, 2020

#68221 has some improvements to the platform bottlenecks we've identified

@rudolf
Copy link
Contributor

rudolf commented Jun 9, 2020

I've created #68626 which, depending on how/where Kibana is deployed, could also increase performance significantly.

@ph
Copy link
Contributor Author

ph commented Jun 25, 2020

@ron @jfsiii I think we can close this issue, with the change to saved object and us moving away from using EQL.

@ph ph closed this as completed Jun 25, 2020
@jfsiii
Copy link
Contributor

jfsiii commented Jun 29, 2020 •
edited
Loading

Getting rid of the KQL filter #69088, #69209, and #69211 saved significant CPU

In the test case from the PRs, total CPU time spent in find dropped from 8 seconds out of ~60 to 0.8 seconds 1.5 seconds and the time in our enroll function went from 4.5 seconds to 0.7 seconds.

@roncohen
Copy link
Contributor

nice work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jfsiii jfsiii

Labels
discuss Team:Fleet Team label for Observability Data Collection Fleet team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ph @roncohen @jfsiii @rudolf @elasticmachine