Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Querier: Refine error messages for per-tenant query limits #5059

Merged
merged 4 commits into from
Jun 8, 2023

Conversation

aknuds1
Copy link
Contributor

@aknuds1 aknuds1 commented May 23, 2023

What this PR does

Refine error messages for per-tenant query limits, informing the user of the preferred strategy for not hitting the limit, in addition to how they may tweak the limit. It's preferred to not increase these limits if possible, since they're designed to prevent Mimir instability. Affected limits:

  • max_fetched_chunks_per_query
  • max_fetched_chunk_bytes_per_query
  • max_fetched_series_per_query

I've taken inspiration from the runbooks for the limits in question on the strategy (same for all three) for not hitting them.

Which issue(s) this PR fixes or relates to

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@aknuds1 aknuds1 added enhancement New feature or request component/querier labels May 23, 2023
@aknuds1 aknuds1 force-pushed the feat/refined-limits-error-messages branch from c831ede to 73fbc91 Compare May 24, 2023 11:11
@aknuds1 aknuds1 force-pushed the feat/refined-limits-error-messages branch from 73fbc91 to 716a111 Compare June 2, 2023 14:28
@flxbk
Copy link
Contributor

flxbk commented Jun 5, 2023

The CHANGELOG has just been cut to prepare for the next Mimir release. Please rebase main and eventually move the CHANGELOG entry added / updated in this PR to the top of the CHANGELOG document. Thanks!

@aknuds1
Copy link
Contributor Author

aknuds1 commented Jun 5, 2023

@pracucci @charleskorn WDYT of my last revision? Query shards no longer mentioned.

Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, modulo failing test

Copy link
Collaborator

@pracucci pracucci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM to me too, thanks

@aknuds1 aknuds1 force-pushed the feat/refined-limits-error-messages branch 3 times, most recently from f10a577 to bb77409 Compare June 8, 2023 06:59
aknuds1 and others added 4 commits June 8, 2023 09:18
Refine error messages for per-tenant query limits, informing the user
of the preferred strategy for not hitting the limit, in addition to how
they may tweak the limit. Affected limits:

* max_fetched_chunks_per_query
* max_fetched_chunk_bytes_per_query
* max_fetched_series_per_query

Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Co-authored-by: Charles Korn <charleskorn@users.noreply.github.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>
@aknuds1 aknuds1 force-pushed the feat/refined-limits-error-messages branch from 305168f to d46ba11 Compare June 8, 2023 07:18
@aknuds1 aknuds1 marked this pull request as ready for review June 8, 2023 07:20
@aknuds1 aknuds1 requested a review from a team as a code owner June 8, 2023 07:20
@aknuds1 aknuds1 enabled auto-merge (squash) June 8, 2023 07:27
@aknuds1 aknuds1 disabled auto-merge June 8, 2023 07:27
@aknuds1 aknuds1 enabled auto-merge (squash) June 8, 2023 07:27
@aknuds1 aknuds1 merged commit ea86996 into main Jun 8, 2023
@aknuds1 aknuds1 deleted the feat/refined-limits-error-messages branch June 8, 2023 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants