autoscaling: add querier predictive scaling #7775
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dimitarvdimitrov
changed the title
autoscaling_querier_predictive_scaling_enabled
charleskorn
reviewed
Apr 2, 2024
dimitarvdimitrov
force-pushed
the
dimitar/jsonnet/querier-predictive-scaling
branch
from
b17cc1a
to
f550086
Compare
Added in a4ba23b |
charleskorn
reviewed
Apr 4, 2024
dimitarvdimitrov
commented
Apr 4, 2024
|
thanks for the review and sorry for the numerous typos 🙈 can you take another pass @charleskorn? |
charleskorn
reviewed
Apr 5, 2024
operations/helm/charts/mimir-distributed/templates/querier/querier-so.yaml
Outdated
Show resolved
Hide resolved
This PR upstreams a change that we have been trialing at GL. ### Problem Some workloads have a very uneven distribution of queries. Some of the causes of this can be external alerting systems and reporting tools which run batch jobs once a day. This results in an influx of queries at fairly regular intervals (~24h). Autoscaling can only respond so fast to incoming queries. In practice, it takes anywhere between 2 and 7 minutes to respond to queries and trigger a scaling event. As a result Mimir's query queues fill up and Mimir start returning HTTP 429 to new queries until queriers have had the time to scale out and accommodate the added load. ### Solution Scale preemptively. Automated workloads are also highly predictive. This PR uses the value of the inflight queries at the same time last week (6d23h30m instead of 7d) to prepare for periodic queries. ### Results We've seen HTTP 429 responses decrease significantly on a weekly basis: between 66% and 95% ### Side effects There are a lot of false-positive scaling events. As a result of this we've seen querier cost increase between 20% and 50%. Given that querier cost is usually 1-2% of the cell's TCO, this is somewhat acceptable for a better QoS. Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
dimitarvdimitrov
force-pushed
the
dimitar/jsonnet/querier-predictive-scaling
branch
from
44b3e3e
to
4db5b1d
Compare
charleskorn
approved these changes
Apr 15, 2024
pracucci
approved these changes
Apr 22, 2024
Signed-off-by: Dimitar Dimitrov <dimitar.dimitrov@grafana.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR upstreams a change that we have been trialing at GL.
Problem
Some workloads have a very uneven distribution of queries. Some of the causes of this can be external alerting systems and reporting tools which run batch jobs once a day.
This results in an influx of queries at fairly regular intervals (~24h). Autoscaling can only respond so fast to incoming queries. In practice, it takes anywhere between 2 and 7 minutes to respond to queries and trigger a scaling event. As a result Mimir's query queues fill up and Mimir start returning HTTP 429 to new queries until queriers have had the time to scale out and accommodate the added load.
Solution
Scale preemptively. Automated workloads are also highly predictive. This PR uses the value of the inflight queries at the same time last week (6d23h30m instead of 7d) to prepare for periodic queries.
Results
We've seen HTTP 429 responses decrease significantly on a weekly basis: between 66% and 95%
Side effects
There are a lot of false-positive scaling events. As a result of this we've seen querier cost increase between 20% and 50%. Given that querier cost is usually 1-2% of the cell's TCO, this is somewhat acceptable for a better QoS.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.