Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cast directly from new PromQL FPoint and HPoint to mimirpb equivalents #4775

Merged
merged 13 commits into from
May 18, 2023

Conversation

zenador
Copy link
Contributor

@zenador zenador commented Apr 19, 2023

What this PR does

Allow casting directly from promql.FPoints to mimirpb.Samples, promql.HPoints to mimirpb.FloatHistogramPairs and histogram.Spans to/from mimirpb.BucketSpans to optimise performance. However, this requires using a pointer in FloatHistogramPair where there wasn't one before, so not sure if we want to keep that part.

Renaming is not done yet. That should probably be done separately, if at all. Not sure if it will be confusing since we've been using Sample for so long, and it will affect much of the codebase, so we need agreement from more people, and maybe also discuss what else we should be renaming for consistency.

Benchmark results here

Which issue(s) this PR fixes or relates to

Fixes #3738 and #4742

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX] (not needed for optimisation)

@zenador zenador requested review from a team as code owners April 19, 2023 11:36
@zenador zenador marked this pull request as draft April 19, 2023 11:36
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch from a199cd9 to bce1bf7 Compare April 19, 2023 18:31
pkg/mimirpb/mimir.proto Outdated Show resolved Hide resolved
pkg/mimirpb/mimir.proto Outdated Show resolved Hide resolved
pkg/util/test/shape.go Outdated Show resolved Hide resolved
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch 3 times, most recently from 6d4bd97 to 916665b Compare April 24, 2023 09:31
pkg/util/test/shape.go Outdated Show resolved Hide resolved
pkg/mimirpb/compat_test.go Outdated Show resolved Hide resolved
pkg/mimirpb/compat_test.go Outdated Show resolved Hide resolved
pkg/mimirpb/compat_test.go Show resolved Hide resolved
pkg/mimirpb/compat_test.go Show resolved Hide resolved
pkg/mimirpb/compat_test.go Show resolved Hide resolved
pkg/mimirpb/compat_test.go Outdated Show resolved Hide resolved
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch from 4da5cfd to d0a8045 Compare April 26, 2023 12:15
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch from c3b3fe9 to c6da0be Compare May 12, 2023 06:21
@zenador zenador marked this pull request as ready for review May 12, 2023 06:21
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch from c6da0be to e81e16f Compare May 12, 2023 09:39
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch from e81e16f to 85b72ec Compare May 16, 2023 08:54
@zenador zenador force-pushed the zenador/align-new-points-for-casting branch from 85b72ec to 6366aab Compare May 16, 2023 09:34
krajorama
krajorama previously approved these changes May 17, 2023
@krajorama krajorama dismissed their stale review May 17, 2023 06:00

olegs comment

@krajorama
Copy link
Contributor

I would suggest to add this small addition to the unit tests so we don't have to rely on the integration tests for catching the loop variable problem:

diff --git a/pkg/frontend/querymiddleware/codec_protobuf_test.go b/pkg/frontend/querymiddleware/codec_protobuf_test.go
index 48ebc7cb9..33647e3d1 100644
--- a/pkg/frontend/querymiddleware/codec_protobuf_test.go
+++ b/pkg/frontend/querymiddleware/codec_protobuf_test.go
@@ -45,6 +45,24 @@ var protobufResponseHistogram = mimirpb.FloatHistogram{
 	PositiveBuckets: []float64{100, 200, 300},
 	NegativeBuckets: []float64{400, 500, 600, 700},
 }
+var protobufResponseHistogram2 = mimirpb.FloatHistogram{
+	CounterResetHint: histogram.GaugeType,
+	Schema:           3,
+	ZeroThreshold:    1.23,
+	ZeroCount:        556,
+	Count:            9101,
+	Sum:              789.1,
+	PositiveSpans: []mimirpb.BucketSpan{
+		{Offset: 4, Length: 1},
+		{Offset: 3, Length: 2},
+	},
+	NegativeSpans: []mimirpb.BucketSpan{
+		{Offset: 7, Length: 3},
+		{Offset: 9, Length: 1},
+	},
+	PositiveBuckets: []float64{100, 200, 300},
+	NegativeBuckets: []float64{400, 500, 600, 700},
+}
 
 var protobufCodecScenarios = []struct {
 	name                  string
@@ -258,6 +276,11 @@ var protobufCodecScenarios = []struct {
 							TimestampMs: 1234,
 							Histogram:   protobufResponseHistogram,
 						},
+						{
+							Metric:      []string{"baz2", "blah2"},
+							TimestampMs: 1234,
+							Histogram:   protobufResponseHistogram2,
+						},
 					},
 				},
 			},
@@ -275,6 +298,10 @@ var protobufCodecScenarios = []struct {
 						Labels:     []mimirpb.LabelAdapter{{Name: "baz", Value: "blah"}},
 						Histograms: []mimirpb.FloatHistogramPair{{TimestampMs: 1234, Histogram: &protobufResponseHistogram}},
 					},
+					{
+						Labels:     []mimirpb.LabelAdapter{{Name: "baz2", Value: "blah2"}},
+						Histograms: []mimirpb.FloatHistogramPair{{TimestampMs: 1234, Histogram: &protobufResponseHistogram2}},
+					},
 				},
 			},
 			Headers: expectedProtobufResponseHeaders,

Copy link
Contributor

@krajorama krajorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@krajorama krajorama merged commit b6e3fa6 into main May 18, 2023
@krajorama krajorama deleted the zenador/align-new-points-for-casting branch May 18, 2023 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Restore mimirpb.FromPointsToSamples performance
4 participants