Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query-tee improvements #8419

Merged
merged 15 commits into from
Jun 21, 2024
Merged

Query-tee improvements #8419

merged 15 commits into from
Jun 21, 2024

Conversation

charleskorn
Copy link
Contributor

@charleskorn charleskorn commented Jun 19, 2024

What this PR does

This PR makes a number of improvements to query-tee. The most notable changes are:

  • Query-tee now emits trace spans when proxying requests: one span is emitted for each incoming request and another for each outgoing request.
  • All log messages associated with a request are tagged with that request's trace ID, just like Mimir itself.
  • Query-tee now correctly configures the Jaeger tracing backend, using the same environment variables as Mimir itself.
  • Log fields such as path, query and route_name are now set on all log messages emitted while processing a request.
  • The user agent is now logged while processing a request.
  • All errors encountered while making a request to a backend are logged, such as DNS failures. (Previously, query-tee relied on the comparator to do this, but the comparator is only used when comparing requests.)
  • Incoming instant query requests without an explicit value for the time parameter now have this explicitly set for outgoing requests to backends, to prevent comparison failures due to different backends using a slightly different timestamp. This is optional, but enabled by default.
  • The microservices Docker Compose environment can now optionally stand up a query-tee instance for testing.

I've tried to make each commit self-contained, and I recommend reviewing each commit separately.

Which issue(s) this PR fixes or relates to

(none)

Checklist

  • Tests updated.
  • [n/a] Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • [n/a] about-versioning.md updated with experimental features.

@charleskorn charleskorn force-pushed the charleskorn/query-tee-improvements branch from e97cc56 to 1d7abaa Compare June 19, 2024 10:29
@charleskorn charleskorn force-pushed the charleskorn/query-tee-improvements branch from 1d7abaa to 5e72af5 Compare June 19, 2024 10:30
@charleskorn charleskorn marked this pull request as ready for review June 19, 2024 10:55
@charleskorn charleskorn requested a review from a team as a code owner June 19, 2024 10:55
Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice stuff, thanks! Lgtm, just nits.

tools/querytee/instant_query_transform.go Outdated Show resolved Hide resolved
"timeout": []string{"60s"},
"time": []string{"2024-06-10T20:30:40Z"},
},
expectedForm: url.Values{},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) could also test where both the form and url params are set

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean time is present in both? Or some parameters are in the URL and some are in the body?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant some parameters are the URL and some in the body. I guess it could also be worth ensuring that if it is set in both that they are passed through unchanged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 527c416.

If time is set in both, then I think all bets are off, so I've left that out for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's set in both I would expect the set values to be passed on. What happens from there I'm unsure, but the former is still something we could check here. Not necessary though imo.

Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. LGTM

@charleskorn charleskorn merged commit cf32b36 into main Jun 21, 2024
29 checks passed
@charleskorn charleskorn deleted the charleskorn/query-tee-improvements branch June 21, 2024 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants