Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming PromQL engine: native histograms #8121

Merged
merged 31 commits into from
May 29, 2024

Conversation

jhesketh
Copy link
Contributor

@jhesketh jhesketh commented May 13, 2024

Add native histogram support to instant vectors.

Checklist

  • Tests updated.
  • Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • about-versioning.md updated with experimental features.

Add native histogram support to instant vectors.
@jhesketh jhesketh marked this pull request as ready for review May 14, 2024 12:31
@jhesketh jhesketh requested a review from a team as a code owner May 14, 2024 12:31
@jhesketh
Copy link
Contributor Author

I've marked this as ready-for-review if we want to put this in in its current state. I'm also happy to keep adding commits to this branch as we support more histogram operations.

Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, thanks for working on this!

Only other thing I'd like to see is some benchmarking:

  • It'd be good to add a benchmark scenario or two that include native histograms
  • It'd be good to compare the performance of the existing non-native histogram benchmarks before and after this change to confirm it doesn't have a significant impact on performance or memory utilisation - I can't imagine it would, but it'd be good to confirm

pkg/streamingpromql/query.go Outdated Show resolved Hide resolved
pkg/streamingpromql/query.go Outdated Show resolved Hide resolved
pkg/streamingpromql/query.go Outdated Show resolved Hide resolved
pkg/streamingpromql/query.go Outdated Show resolved Hide resolved
pkg/streamingpromql/testdata/ours/native_histograms.test Outdated Show resolved Hide resolved
pkg/streamingpromql/operator/instant_vector_selector.go Outdated Show resolved Hide resolved
pkg/streamingpromql/operator/instant_vector_selector.go Outdated Show resolved Hide resolved
@jhesketh
Copy link
Contributor Author

jhesketh commented May 28, 2024

Benchmark of this branch in details:

goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
cpu: 12th Gen Intel(R) Core(TM) i7-1270P
                                                                       │      Prometheus      │              streaming              │
                                                                       │        sec/op        │    sec/op     vs base               │
Query/a_1,_instant_query                                                         148.0µ ±  4%   133.4µ ±  6%   -9.88% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                            158.1µ ±  3%   146.0µ ±  1%   -7.66% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                           260.2µ ±  1%   242.8µ ±  4%   -6.66% (p=0.002 n=6)
Query/a_100,_instant_query                                                       1.071m ±  2%   1.047m ±  1%   -2.22% (p=0.015 n=6)
Query/a_100,_range_query_with_100_steps                                          2.301m ± 22%   1.798m ±  2%  -21.86% (p=0.009 n=6)
Query/a_100,_range_query_with_1000_steps                                         9.551m ±  1%   9.691m ±  3%   +1.47% (p=0.026 n=6)
Query/a_2000,_instant_query                                                      18.54m ±  1%   18.31m ±  3%        ~ (p=0.132 n=6)
Query/a_2000,_range_query_with_100_steps                                         32.44m ±  1%   32.01m ±  1%        ~ (p=0.065 n=6)
Query/a_2000,_range_query_with_1000_steps                                        156.2m ±  2%   155.4m ±  2%   -0.51% (p=0.041 n=6)
Query/nh_1,_instant_query                                                        246.6µ ±  7%   226.4µ ±  4%   -8.16% (p=0.002 n=6)
Query/nh_1,_range_query_with_100_steps                                           328.0µ ±  2%   319.7µ ±  5%        ~ (p=0.240 n=6)
Query/nh_1,_range_query_with_1000_steps                                          775.3µ ±  1%   953.6µ ± 20%        ~ (p=0.065 n=6)
Query/nh_100,_instant_query                                                      2.404m ±  2%   2.369m ±  2%        ~ (p=0.065 n=6)
Query/nh_100,_range_query_with_100_steps                                         9.274m ± 19%   7.308m ±  2%  -21.19% (p=0.002 n=6)
Query/nh_100,_range_query_with_1000_steps                                        57.76m ±  3%   58.53m ±  4%        ~ (p=0.589 n=6)
Query/nh_2000,_instant_query                                                     46.69m ±  2%   46.17m ±  3%        ~ (p=0.310 n=6)
Query/nh_2000,_range_query_with_100_steps                                        146.2m ±  3%   147.8m ±  2%        ~ (p=0.818 n=6)
Query/nh_2000,_range_query_with_1000_steps                                        1.309 ±  2%    1.289 ±  2%   -1.56% (p=0.026 n=6)
Query/a_1[1m],_instant_query                                                     203.8µ ±  3%   180.5µ ±  6%  -11.41% (p=0.002 n=6)
Query/a_100[1m],_instant_query                                                   583.6µ ±  2%   572.6µ ±  1%   -1.89% (p=0.002 n=6)
Query/a_2000[1m],_instant_query                                                  6.755m ±  3%   5.729m ±  1%  -15.19% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                               205.1µ ±  3%   184.2µ ±  4%  -10.16% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                  233.1µ ±  5%   200.7µ ±  5%  -13.91% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                 426.0µ ±  3%   338.0µ ±  3%  -20.66% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                             765.8µ ±  2%   579.7µ ±  3%  -24.30% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                2.519m ±  3%   1.764m ±  2%  -29.97% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                               16.34m ±  3%   10.33m ±  2%  -36.77% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                            9.358m ±  2%   6.639m ±  1%  -29.05% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                               34.88m ± 13%   25.84m ±  2%  -25.92% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                              253.8m ±  1%   157.4m ±  3%  -37.99% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                1.963m ±  2%   1.421m ±  1%  -27.60% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                             128.81m ±  2%   81.30m ±  1%  -36.89% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                              2.726 ±  1%    1.630 ±  2%  -40.21% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                               1.441m ±  3%   1.099m ±  2%  -23.75% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                  2.120m ±  2%   1.598m ±  2%  -24.65% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                 7.155m ±  2%   5.781m ±  2%  -19.20% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                             64.16m ±  2%   54.78m ±  2%  -14.62% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                118.5m ±  1%   100.0m ±  2%  -15.61% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                               577.0m ±  2%   504.2m ±  2%  -12.61% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                           1133.0m ±  3%   959.7m ±  1%  -15.30% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                                2.197 ±  1%    1.853 ±  2%  -15.67% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                              11.433 ±  1%    9.840 ±  1%  -13.93% (p=0.002 n=6)
Query/a_1_-_b_1,_instant_query                                                   357.2µ ±  3%   326.6µ ±  6%   -8.55% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                      407.2µ ±  1%   349.0µ ±  2%  -14.29% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                     891.0µ ±  3%   559.4µ ±  2%  -37.21% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                               2.338m ±  1%   2.323m ±  2%        ~ (p=0.180 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                  6.159m ±  2%   3.826m ±  2%  -37.89% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                 36.81m ±  1%   16.17m ±  4%  -56.09% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                             33.00m ±  2%   32.07m ±  2%   -2.81% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                               106.58m ±  1%   56.55m ±  1%  -46.94% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                               818.0m ±  1%   276.8m ±  2%  -66.16% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    4.875m ±  2%   2.415m ±  1%  -50.46% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                                338.5m ±  2%   134.8m ±  1%  -60.17% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                               8.693 ±  0%    2.675 ±  1%  -69.23% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                2.219m ±  2%   2.204m ±  3%        ~ (p=0.394 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                   4.818m ±  3%   3.377m ±  1%  -29.90% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                  26.86m ±  3%   14.42m ±  2%  -46.32% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                  25.30m ±  2%   21.53m ±  1%  -14.90% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                     72.59m ±  4%   36.64m ±  1%  -49.53% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                    517.3m ±  2%   173.2m ±  2%  -66.52% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                    181.9µ ±  2%   162.6µ ±  1%  -10.63% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                       202.3µ ±  1%   177.8µ ±  2%  -12.10% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                      345.1µ ±  3%   295.1µ ±  2%  -14.50% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                  1.193m ±  2%   1.158m ±  1%   -2.98% (p=0.004 n=6)
Query/sum(a_100),_range_query_with_100_steps                                     1.989m ±  2%   1.925m ±  2%   -3.18% (p=0.004 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                    8.729m ±  5%   8.543m ±  2%   -2.14% (p=0.009 n=6)
Query/sum(a_2000),_instant_query                                                 16.12m ±  3%   15.73m ±  3%   -2.42% (p=0.041 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                    29.80m ±  2%   28.41m ±  2%   -4.68% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                   153.5m ±  2%   143.7m ±  3%   -6.41% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                             249.1µ ±  4%   236.5µ ±  4%   -5.07% (p=0.009 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                316.3µ ±  2%   292.5µ ±  3%   -7.51% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                               874.3µ ±  2%   813.3µ ±  2%   -6.98% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                           5.650m ±  1%   5.559m ±  1%   -1.62% (p=0.015 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                             10.083m ±  2%   9.740m ±  1%   -3.40% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                             51.21m ±  2%   46.22m ±  2%   -9.75% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                          96.32m ±  2%   95.69m ±  3%        ~ (p=0.180 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                             177.6m ±  2%   169.4m ±  1%   -4.63% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                           1154.6m ±  2%   870.4m ±  1%  -24.61% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                            250.0µ ±  3%   231.1µ ±  2%   -7.55% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                               319.4µ ±  3%   301.1µ ±  1%   -5.73% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                              900.7µ ±  1%   840.5µ ±  1%   -6.67% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                          5.448m ±  1%   5.335m ±  2%   -2.06% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                             9.671m ±  1%   9.334m ±  2%   -3.48% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                            48.98m ±  2%   46.28m ±  1%   -5.51% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                         95.01m ±  1%   91.96m ±  1%   -3.21% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                            177.5m ±  2%   166.3m ±  2%   -6.28% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                          1106.8m ±  1%   857.5m ±  4%  -22.52% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                               355.9µ ±  2%   316.2µ ±  3%  -11.16% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                  433.8µ ±  1%   349.7µ ±  2%  -19.39% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                1002.2µ ±  2%   600.3µ ±  1%  -40.11% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                           1.460m ±  2%   1.134m ±  1%  -22.31% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps              6.636m ±  1%   3.205m ±  2%  -51.71% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps             48.90m ±  1%   18.59m ±  1%  -61.99% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                         17.70m ±  1%   13.02m ±  3%  -26.45% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps           118.20m ±  2%   45.63m ±  5%  -61.40% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps          1083.7m ±  2%   317.8m ±  2%  -70.67% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                              366.7µ ±  2%   336.7µ ±  3%   -8.19% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                 422.4µ ±  2%   356.8µ ±  2%  -15.52% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                                945.3µ ±  2%   581.9µ ±  2%  -38.44% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                          2.347m ±  1%   2.296m ±  1%   -2.16% (p=0.009 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                             6.101m ±  1%   3.793m ±  3%  -37.83% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                            37.57m ±  2%   16.63m ±  2%  -55.74% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                        33.04m ±  2%   31.80m ±  2%   -3.76% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                          107.76m ±  1%   57.14m ±  1%  -46.97% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                          840.0m ±  1%   281.6m ±  1%  -66.48% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                  224.8µ ±  2%   191.8µ ±  6%  -14.70% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                     358.3µ ±  2%   274.4µ ±  1%  -23.43% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                   1322.5µ ±  2%   925.7µ ±  1%  -30.00% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                3.075m ±  1%   2.219m ±  1%  -27.83% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                  11.788m ±  1%   7.754m ±  2%  -34.22% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                  85.07m ±  2%   52.19m ±  2%  -38.65% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                               49.76m ±  2%   35.74m ±  2%  -28.18% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                  214.0m ±  1%   137.1m ±  3%  -35.92% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                  1.812 ±  1%    1.000 ±  2%  -44.81% (p=0.002 n=6)
geomean                                                                          10.33m         7.892m        -23.59%

                                                                       │      Prometheus      │               streaming               │
                                                                       │         B/op         │      B/op       vs base               │
Query/a_1,_instant_query                                                         20.68Ki ± 0%    17.75Ki ±  0%  -14.16% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                            21.60Ki ± 0%    18.32Ki ±  0%  -15.19% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                           26.19Ki ± 0%    22.88Ki ±  0%  -12.65% (p=0.002 n=6)
Query/a_100,_instant_query                                                       169.6Ki ± 0%    151.8Ki ±  0%  -10.53% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                          222.4Ki ± 0%    208.6Ki ±  0%   -6.20% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                         658.3Ki ± 0%    645.4Ki ±  0%   -1.96% (p=0.002 n=6)
Query/a_2000,_instant_query                                                      3.019Mi ± 0%    2.690Mi ±  0%  -10.89% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                         3.965Mi ± 0%    3.771Mi ±  1%   -4.90% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                        12.34Mi ± 0%    12.14Mi ±  0%   -1.59% (p=0.002 n=6)
Query/nh_1,_instant_query                                                        37.10Ki ± 0%    34.09Ki ±  0%   -8.10% (p=0.002 n=6)
Query/nh_1,_range_query_with_100_steps                                           88.56Ki ± 0%    85.46Ki ±  0%   -3.50% (p=0.002 n=6)
Query/nh_1,_range_query_with_1000_steps                                          520.0Ki ± 0%    517.4Ki ±  0%   -0.48% (p=0.002 n=6)
Query/nh_100,_instant_query                                                      940.2Ki ± 0%    922.1Ki ±  0%   -1.93% (p=0.002 n=6)
Query/nh_100,_range_query_with_100_steps                                         5.568Mi ± 0%    5.563Mi ±  0%   -0.10% (p=0.002 n=6)
Query/nh_100,_range_query_with_1000_steps                                        47.51Mi ± 0%    47.52Mi ±  0%        ~ (p=0.818 n=6)
Query/nh_2000,_instant_query                                                     17.76Mi ± 0%    17.46Mi ±  0%   -1.68% (p=0.002 n=6)
Query/nh_2000,_range_query_with_100_steps                                        111.6Mi ± 0%    111.5Mi ±  0%        ~ (p=0.240 n=6)
Query/nh_2000,_range_query_with_1000_steps                                       977.2Mi ± 0%    976.8Mi ±  0%   -0.05% (p=0.004 n=6)
Query/a_1[1m],_instant_query                                                     21.19Ki ± 0%    17.38Ki ±  0%  -17.98% (p=0.002 n=6)
Query/a_100[1m],_instant_query                                                   132.7Ki ± 0%    109.2Ki ±  0%  -17.69% (p=0.002 n=6)
Query/a_2000[1m],_instant_query                                                  2.291Mi ± 0%    1.903Mi ±  0%  -16.92% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                               22.47Ki ± 0%    17.82Ki ±  0%  -20.70% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                  23.40Ki ± 1%    18.25Ki ±  0%  -21.98% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                 28.21Ki ± 1%    22.90Ki ±  0%  -18.82% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                             151.8Ki ± 0%    111.5Ki ±  0%  -26.51% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                192.2Ki ± 0%    155.6Ki ±  0%  -19.05% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                               651.5Ki ± 0%    613.3Ki ±  0%   -5.85% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                            2.616Mi ± 0%    1.878Mi ±  0%  -28.21% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                               3.381Mi ± 0%    2.794Mi ±  1%  -17.36% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                              12.08Mi ± 0%    11.51Mi ±  1%   -4.75% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                88.98Ki ± 6%    71.83Ki ±  1%  -19.26% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                              5.204Mi ± 3%    5.021Mi ±  2%   -3.53% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                            1069.4Mi ± 0%    600.0Mi ±  0%  -43.89% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                              669.59Ki ± 1%    71.99Ki ±  2%  -89.25% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                 671.67Ki ± 2%    71.70Ki ±  6%  -89.33% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                630.42Ki ± 4%    73.71Ki ± 11%  -88.31% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                             5.267Mi ± 2%    4.723Mi ±  2%  -10.32% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                5.311Mi ± 3%    4.719Mi ±  4%  -11.15% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                               5.767Mi ± 8%    5.374Mi ±  8%   -6.83% (p=0.026 n=6)
Query/rate(a_2000[1d]),_instant_query                                            90.78Mi ± 1%    88.04Mi ±  1%   -3.02% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                               94.53Mi ± 1%    94.14Mi ±  1%        ~ (p=0.180 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                              128.8Mi ± 0%    128.1Mi ±  0%   -0.57% (p=0.009 n=6)
Query/a_1_-_b_1,_instant_query                                                   42.60Ki ± 0%    36.46Ki ±  0%  -14.40% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                      48.87Ki ± 0%    37.57Ki ±  0%  -23.11% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                    100.78Ki ± 0%    46.61Ki ±  0%  -53.75% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                               473.9Ki ± 0%    338.6Ki ±  0%  -28.55% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                  585.8Ki ± 0%    452.7Ki ±  0%  -22.72% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                 1.472Mi ± 0%    1.296Mi ±  0%  -11.96% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                             8.178Mi ± 0%    5.940Mi ±  0%  -27.37% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                               10.218Mi ± 0%    8.094Mi ±  0%  -20.78% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                               27.94Mi ± 1%    24.82Mi ±  1%  -11.18% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    690.2Ki ± 3%    143.5Ki ±  4%  -79.21% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                               10.689Mi ± 2%    9.966Mi ±  1%   -6.76% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                             3108.2Mi ± 0%    700.4Mi ±  0%  -77.47% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                373.2Ki ± 0%    256.7Ki ±  0%  -31.21% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                   455.5Ki ± 0%    339.3Ki ±  0%  -25.52% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                 1159.1Ki ± 0%    999.3Ki ±  0%  -13.79% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                  6.160Mi ± 0%    3.930Mi ±  0%  -36.20% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                     7.703Mi ± 0%    5.339Mi ±  1%  -30.68% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                    20.74Mi ± 1%    16.40Mi ±  0%  -20.90% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                    21.80Ki ± 0%    18.33Ki ±  0%  -15.90% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                       22.79Ki ± 1%    20.00Ki ±  0%  -12.24% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                      27.42Ki ± 0%    32.59Ki ±  0%  +18.88% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                  174.2Ki ± 0%    153.4Ki ±  0%  -11.96% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                                     230.7Ki ± 0%    210.9Ki ±  0%   -8.57% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                    665.7Ki ± 0%    658.4Ki ±  0%   -1.10% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                                 3.082Mi ± 0%    2.677Mi ±  0%  -13.16% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                    4.109Mi ± 0%    3.711Mi ±  0%   -9.69% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                   12.48Mi ± 0%    12.24Mi ±  1%   -1.92% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                             29.75Ki ± 1%    25.34Ki ±  0%  -14.82% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                33.43Ki ± 1%    29.69Ki ±  0%  -11.19% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                               60.69Ki ± 0%    64.84Ki ±  0%   +6.83% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                          1120.5Ki ± 0%    977.8Ki ±  0%  -12.73% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                              1.399Mi ± 0%    1.374Mi ±  0%   -1.79% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                             3.926Mi ± 1%    4.722Mi ±  0%  +20.29% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                          22.49Mi ± 0%    19.57Mi ±  0%  -12.98% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                             28.32Mi ± 0%    29.07Mi ±  1%   +2.62% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                            298.8Mi ± 0%    105.6Mi ±  4%  -64.66% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                            31.20Ki ± 0%    25.50Ki ±  0%  -18.28% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                               34.77Ki ± 1%    35.28Ki ±  0%   +1.46% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                              62.03Ki ± 0%   110.46Ki ±  0%  +78.09% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                         1079.9Ki ± 0%    949.4Ki ±  0%  -12.08% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                             1.365Mi ± 0%    1.248Mi ±  0%   -8.54% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                            3.891Mi ± 1%    3.852Mi ±  0%   -1.00% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                         21.67Mi ± 0%    19.03Mi ±  0%  -12.20% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                            27.63Mi ± 0%    25.26Mi ±  0%   -8.58% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                          266.90Mi ± 0%    76.10Mi ±  0%  -71.49% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                               46.26Ki ± 0%    36.54Ki ±  0%  -21.01% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                  52.31Ki ± 0%    37.42Ki ±  0%  -28.47% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                104.59Ki ± 0%    46.63Ki ±  0%  -55.41% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                           431.9Ki ± 0%    253.0Ki ±  0%  -41.43% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps              519.6Ki ± 1%    341.5Ki ±  0%  -34.28% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps             1.448Mi ± 0%    1.229Mi ±  0%  -15.09% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                         7.275Mi ± 0%    4.233Mi ±  0%  -41.81% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps            8.953Mi ± 1%    6.031Mi ±  1%  -32.64% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps          121.77Mi ± 0%    23.45Mi ±  2%  -80.74% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                              43.92Ki ± 0%    37.09Ki ±  0%  -15.56% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                 50.17Ki ± 1%    39.29Ki ±  0%  -21.68% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                               102.10Ki ± 1%    56.42Ki ±  0%  -44.74% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                          478.6Ki ± 0%    340.4Ki ±  0%  -28.88% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                             595.3Ki ± 0%    453.9Ki ±  0%  -23.76% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                            1.479Mi ± 0%    1.309Mi ±  0%  -11.48% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                        8.249Mi ± 0%    5.909Mi ±  0%  -28.37% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                          10.358Mi ± 0%    8.020Mi ±  1%  -22.57% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                          28.06Mi ± 0%    25.05Mi ±  1%  -10.74% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                  32.43Ki ± 0%    23.42Ki ±  0%  -27.76% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                     34.89Ki ± 1%    32.49Ki ±  0%   -6.88% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                    62.62Ki ± 0%   108.41Ki ±  0%  +73.13% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                953.3Ki ± 0%    707.4Ki ±  0%  -25.79% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                  1216.0Ki ± 0%    977.7Ki ±  0%  -19.60% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                  3.859Mi ± 0%    3.654Mi ±  0%   -5.32% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                               19.17Mi ± 0%    14.16Mi ±  0%  -26.09% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                  24.05Mi ± 0%    19.41Mi ±  1%  -19.29% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                265.37Mi ± 0%    73.03Mi ±  1%  -72.48% (p=0.002 n=6)
geomean                                                                          1.242Mi         962.2Ki        -24.34%

                                                                       │      Prometheus      │             streaming              │
                                                                       │      allocs/op       │  allocs/op   vs base               │
Query/a_1,_instant_query                                                           370.0 ± 0%    313.0 ± 0%  -15.41% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                              380.0 ± 0%    317.0 ± 0%  -16.58% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                             412.0 ± 0%    349.0 ± 0%  -15.29% (p=0.002 n=6)
Query/a_100,_instant_query                                                        2.287k ± 0%   2.222k ± 0%   -2.84% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                           2.700k ± 0%   2.630k ± 0%   -2.59% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                          5.907k ± 0%   5.838k ± 0%   -1.16% (p=0.002 n=6)
Query/a_2000,_instant_query                                                       38.89k ± 0%   38.80k ± 0%   -0.24% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                          46.92k ± 0%   46.84k ± 0%   -0.16% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                         110.9k ± 0%   110.9k ± 0%   -0.07% (p=0.002 n=6)
Query/nh_1,_instant_query                                                          638.0 ± 0%    581.0 ± 0%   -8.93% (p=0.002 n=6)
Query/nh_1,_range_query_with_100_steps                                            1.716k ± 0%   1.653k ± 0%   -3.67% (p=0.002 n=6)
Query/nh_1,_range_query_with_1000_steps                                           10.79k ± 0%   10.73k ± 0%   -0.57% (p=0.002 n=6)
Query/nh_100,_instant_query                                                       19.20k ± 0%   19.14k ± 0%   -0.36% (p=0.002 n=6)
Query/nh_100,_range_query_with_100_steps                                          120.1k ± 0%   120.0k ± 0%   -0.05% (p=0.002 n=6)
Query/nh_100,_range_query_with_1000_steps                                         1.027M ± 0%   1.027M ± 0%   -0.01% (p=0.002 n=6)
Query/nh_2000,_instant_query                                                      375.7k ± 0%   375.0k ± 0%   -0.17% (p=0.002 n=6)
Query/nh_2000,_range_query_with_100_steps                                         2.393M ± 0%   2.393M ± 0%   -0.01% (p=0.015 n=6)
Query/nh_2000,_range_query_with_1000_steps                                        20.54M ± 0%   20.54M ± 0%   -0.00% (p=0.002 n=6)
Query/a_1[1m],_instant_query                                                       380.0 ± 0%    310.0 ± 0%  -18.42% (p=0.002 n=6)
Query/a_100[1m],_instant_query                                                    1.903k ± 0%   1.819k ± 0%   -4.41% (p=0.002 n=6)
Query/a_2000[1m],_instant_query                                                   30.85k ± 0%   30.76k ± 0%   -0.30% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                                 409.0 ± 0%    320.0 ± 0%  -21.76% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                    419.0 ± 0%    324.0 ± 0%  -22.67% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                   452.0 ± 0%    357.0 ± 0%  -21.02% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                              2.324k ± 0%   1.927k ± 0%  -17.08% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                 2.736k ± 0%   2.334k ± 0%  -14.69% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                                6.051k ± 0%   5.650k ± 0%   -6.63% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                             38.88k ± 0%   32.75k ± 0%  -15.75% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                                46.95k ± 0%   40.84k ± 0%  -13.01% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                               113.0k ± 0%   106.9k ± 0%   -5.42% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                  761.0 ± 0%    665.0 ± 0%  -12.61% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                               36.41k ± 0%   36.01k ± 0%   -1.11% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                              732.3k ± 0%   720.0k ± 0%   -1.68% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                                 724.0 ± 0%    618.0 ± 0%  -14.64% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                    732.5 ± 0%    621.5 ± 0%  -15.15% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                   760.0 ± 0%    653.0 ± 0%  -14.08% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                              32.26k ± 0%   31.83k ± 0%   -1.33% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                 32.56k ± 0%   32.15k ± 0%   -1.26% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                                35.59k ± 0%   35.21k ± 0%   -1.05% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                             641.9k ± 0%   629.9k ± 0%   -1.86% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                                648.0k ± 0%   641.8k ± 0%   -0.95% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                               708.2k ± 0%   702.1k ± 0%   -0.86% (p=0.002 n=6)
Query/a_1_-_b_1,_instant_query                                                     740.0 ± 0%    645.0 ± 0%  -12.84% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                        952.0 ± 0%    653.0 ± 0%  -31.41% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                      2817.0 ± 0%    718.0 ± 0%  -74.51% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                                5.198k ± 0%   5.270k ± 0%   +1.39% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                   6.016k ± 0%   6.085k ± 0%   +1.16% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                  14.24k ± 0%   12.50k ± 0%  -12.17% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                              89.80k ± 0%   93.61k ± 0%   +4.25% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                                 102.1k ± 0%   109.7k ± 0%   +7.48% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                                232.0k ± 0%   237.8k ± 0%   +2.48% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    21.431k ± 0%   1.328k ± 0%  -93.80% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                                 92.34k ± 0%   72.61k ± 0%  -21.37% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                               1.484M ± 0%   1.446M ± 0%   -2.58% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                 4.044k ± 0%   4.015k ± 0%   -0.72% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                    4.711k ± 0%   4.629k ± 0%   -1.74% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                  11.329k ± 0%   9.447k ± 0%  -16.61% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                   66.59k ± 0%   58.17k ± 0%  -12.64% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                      75.81k ± 0%   67.35k ± 0%  -11.16% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                     173.7k ± 0%   136.4k ± 0%  -21.47% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                      394.0 ± 0%    322.0 ± 0%  -18.27% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                         404.0 ± 0%    330.0 ± 0%  -18.32% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                        436.0 ± 0%    362.0 ± 0%  -16.97% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                   2.312k ± 0%   2.232k ± 0%   -3.46% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                                      2.724k ± 0%   2.643k ± 0%   -2.97% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                     5.931k ± 0%   5.849k ± 0%   -1.37% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                                  38.92k ± 0%   38.82k ± 0%   -0.26% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                     46.94k ± 0%   46.84k ± 0%   -0.21% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                    111.0k ± 0%   110.9k ± 0%   -0.04% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                               507.0 ± 0%    429.0 ± 0%  -15.38% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                  537.0 ± 0%    457.0 ± 0%  -14.90% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                                 729.0 ± 0%    650.0 ± 0%  -10.84% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                            12.69k ± 0%   12.59k ± 0%   -0.77% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                               15.10k ± 0%   15.41k ± 0%   +2.09% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                              34.32k ± 0%   34.66k ± 0%   +0.99% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                           260.1k ± 1%   246.4k ± 0%   -5.25% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                              293.5k ± 0%   303.8k ± 0%   +3.53% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                             707.6k ± 0%   691.2k ± 0%   -2.33% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                              515.0 ± 0%    434.0 ± 0%  -15.73% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                                 545.0 ± 0%    483.0 ± 0%  -11.38% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                                737.0 ± 0%    675.0 ± 0%   -8.41% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                           12.57k ± 0%   12.48k ± 0%   -0.73% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                              14.98k ± 0%   14.91k ± 0%   -0.48% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                             34.21k ± 0%   34.15k ± 0%   -0.17% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                          255.3k ± 1%   243.3k ± 0%   -4.68% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                             291.4k ± 0%   291.4k ± 0%        ~ (p=0.390 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                            701.6k ± 0%   675.7k ± 0%   -3.69% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                                 810.0 ± 0%    657.0 ± 0%  -18.89% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                   1023.0 ± 0%    665.0 ± 0%  -35.00% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                  2889.0 ± 0%    731.0 ± 0%  -74.70% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                            5.268k ± 0%   4.681k ± 0%  -11.14% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps               6.084k ± 0%   5.494k ± 0%   -9.70% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps              14.52k ± 0%   12.13k ± 0%  -16.47% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                          89.80k ± 0%   81.57k ± 0%   -9.17% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps            102.07k ± 0%   97.66k ± 0%   -4.32% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps            250.0k ± 0%   229.8k ± 0%   -8.08% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                                764.0 ± 0%    655.0 ± 0%  -14.27% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                   976.0 ± 0%    667.0 ± 0%  -31.66% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                                 2841.0 ± 0%    731.0 ± 0%  -74.27% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                           5.223k ± 0%   5.281k ± 0%   +1.12% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                              6.040k ± 0%   6.098k ± 0%   +0.96% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                             14.26k ± 0%   12.52k ± 0%  -12.21% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                         89.83k ± 0%   93.63k ± 0%   +4.24% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                            102.1k ± 0%   109.7k ± 0%   +7.45% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                           232.1k ± 0%   237.9k ± 0%   +2.49% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                    555.0 ± 0%    420.0 ± 0%  -24.32% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                       577.0 ± 0%    469.0 ± 0%  -18.72% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                      776.0 ± 0%    668.0 ± 0%  -13.92% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                 12.61k ± 0%   10.67k ± 0%  -15.34% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                    15.02k ± 0%   13.12k ± 0%  -12.69% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                   34.87k ± 0%   32.97k ± 0%   -5.46% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                                252.4k ± 1%   207.0k ± 0%  -17.97% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                   291.5k ± 0%   255.4k ± 0%  -12.40% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                  713.7k ± 0%   652.6k ± 0%   -8.56% (p=0.002 n=6)
geomean                                                                           13.41k        11.62k       -13.35%

                                                                       │      Prometheus      │              streaming               │
                                                                       │          B           │       B        vs base               │
Query/a_1,_instant_query                                                         74.50Ki ± 1%   73.19Ki ±  1%   -1.76% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                            74.31Ki ± 1%   72.93Ki ±  2%   -1.86% (p=0.039 n=6)
Query/a_1,_range_query_with_1000_steps                                           72.49Ki ± 2%   71.19Ki ±  2%   -1.81% (p=0.026 n=6)
Query/a_100,_instant_query                                                       67.75Ki ± 2%   67.16Ki ±  1%        ~ (p=0.143 n=6)
Query/a_100,_range_query_with_100_steps                                          67.39Ki ± 2%   67.35Ki ±  2%        ~ (p=0.937 n=6)
Query/a_100,_range_query_with_1000_steps                                         69.89Ki ± 1%   69.30Ki ±  1%        ~ (p=0.394 n=6)
Query/a_2000,_instant_query                                                      68.68Ki ± 1%   68.36Ki ±  1%        ~ (p=0.180 n=6)
Query/a_2000,_range_query_with_100_steps                                         75.65Ki ± 2%   76.92Ki ±  1%   +1.68% (p=0.004 n=6)
Query/a_2000,_range_query_with_1000_steps                                        135.6Ki ± 1%   136.0Ki ±  1%        ~ (p=0.167 n=6)
Query/nh_1,_instant_query                                                        82.00Ki ± 3%   82.31Ki ±  1%        ~ (p=0.574 n=6)
Query/nh_1,_range_query_with_100_steps                                           74.45Ki ± 2%   74.19Ki ±  2%        ~ (p=0.485 n=6)
Query/nh_1,_range_query_with_1000_steps                                          72.19Ki ± 2%   71.07Ki ±  1%        ~ (p=0.065 n=6)
Query/nh_100,_instant_query                                                      68.68Ki ± 2%   68.63Ki ±  1%        ~ (p=0.818 n=6)
Query/nh_100,_range_query_with_100_steps                                         73.36Ki ± 1%   73.59Ki ±  1%        ~ (p=0.725 n=6)
Query/nh_100,_range_query_with_1000_steps                                        118.4Ki ± 0%   118.2Ki ±  0%        ~ (p=0.699 n=6)
Query/nh_2000,_instant_query                                                     70.72Ki ± 2%   70.27Ki ±  1%        ~ (p=0.485 n=6)
Query/nh_2000,_range_query_with_100_steps                                        172.6Ki ± 1%   174.6Ki ±  1%   +1.16% (p=0.002 n=6)
Query/nh_2000,_range_query_with_1000_steps                                       706.8Ki ± 3%   715.1Ki ±  4%        ~ (p=1.000 n=6)
Query/a_1[1m],_instant_query                                                     73.56Ki ± 2%   72.56Ki ±  1%        ~ (p=0.331 n=6)
Query/a_100[1m],_instant_query                                                   67.68Ki ± 2%   67.46Ki ±  1%        ~ (p=0.589 n=6)
Query/a_2000[1m],_instant_query                                                  70.67Ki ± 1%   70.34Ki ±  1%        ~ (p=0.093 n=6)
Query/rate(a_1[1m]),_instant_query                                               72.84Ki ± 1%   72.88Ki ±  3%        ~ (p=0.851 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                  72.69Ki ± 1%   73.07Ki ±  2%        ~ (p=0.485 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                 70.75Ki ± 1%   70.72Ki ±  2%        ~ (p=0.788 n=6)
Query/rate(a_100[1m]),_instant_query                                             67.58Ki ± 1%   67.64Ki ±  2%        ~ (p=0.818 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                67.50Ki ± 2%   66.71Ki ±  2%        ~ (p=0.065 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                               68.62Ki ± 1%   69.48Ki ±  1%   +1.25% (p=0.004 n=6)
Query/rate(a_2000[1m]),_instant_query                                            69.27Ki ± 2%   69.42Ki ±  1%        ~ (p=0.699 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                               75.45Ki ± 1%   76.88Ki ±  1%   +1.89% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                              130.6Ki ± 1%   131.8Ki ±  1%   +0.99% (p=0.004 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                69.11Ki ± 2%   69.33Ki ±  1%        ~ (p=0.675 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                              100.5Ki ± 0%   108.8Ki ±  1%   +8.24% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                             547.8Ki ± 5%   426.8Ki ±  2%  -22.08% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                               68.60Ki ± 1%   73.50Ki ±  1%   +7.14% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                  68.46Ki ± 1%   71.72Ki ±  2%   +4.76% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                 67.25Ki ± 1%   68.94Ki ±  2%   +2.50% (p=0.017 n=6)
Query/rate(a_100[1d]),_instant_query                                             71.87Ki ± 2%   73.06Ki ±  2%        ~ (p=0.126 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                70.94Ki ± 2%   73.03Ki ±  5%        ~ (p=0.104 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                               66.31Ki ± 2%   65.94Ki ±  7%        ~ (p=0.931 n=6)
Query/rate(a_2000[1d]),_instant_query                                            78.38Ki ± 3%   79.86Ki ±  2%        ~ (p=0.093 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                               80.65Ki ± 2%   81.81Ki ±  2%        ~ (p=0.180 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                              113.8Ki ± 3%   113.3Ki ±  3%        ~ (p=0.461 n=6)
Query/a_1_-_b_1,_instant_query                                                   72.62Ki ± 1%   73.19Ki ±  2%        ~ (p=0.667 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                      71.94Ki ± 1%   72.00Ki ±  2%        ~ (p=0.519 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                     69.89Ki ± 1%   70.29Ki ±  1%        ~ (p=0.310 n=6)
Query/a_100_-_b_100,_instant_query                                               67.86Ki ± 1%   67.59Ki ±  1%        ~ (p=0.132 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                  68.24Ki ± 1%   68.00Ki ±  1%        ~ (p=0.394 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                 75.61Ki ± 2%   71.00Ki ±  1%   -6.10% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                             70.00Ki ± 1%   71.12Ki ±  1%   +1.60% (p=0.026 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                                93.87Ki ± 1%   80.00Ki ±  1%  -14.77% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                               207.9Ki ± 1%   142.2Ki ±  2%  -31.62% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    69.82Ki ± 1%   69.91Ki ±  1%        ~ (p=0.818 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                                141.7Ki ± 2%   119.8Ki ±  3%  -15.50% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                             1615.4Ki ± 2%   459.0Ki ±  3%  -71.59% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                67.64Ki ± 2%   67.81Ki ±  2%        ~ (p=0.851 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                   67.87Ki ± 2%   67.42Ki ±  1%        ~ (p=0.240 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                  71.93Ki ± 2%   68.05Ki ±  1%   -5.40% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                  70.06Ki ± 2%   97.06Ki ±  3%  +38.54% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                     84.11Ki ± 1%   99.19Ki ± 32%  +17.93% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                    160.4Ki ± 0%   124.4Ki ±  8%  -22.45% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                    73.88Ki ± 2%   73.06Ki ±  3%        ~ (p=0.071 n=6)
Query/sum(a_1),_range_query_with_100_steps                                       72.75Ki ± 1%   74.10Ki ±  2%   +1.86% (p=0.009 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                      71.67Ki ± 1%   78.88Ki ±  2%  +10.06% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                  66.99Ki ± 1%   67.00Ki ±  1%        ~ (p=0.937 n=6)
Query/sum(a_100),_range_query_with_100_steps                                     68.25Ki ± 1%   66.89Ki ±  1%   -1.99% (p=0.026 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                    69.79Ki ± 2%   67.47Ki ±  0%   -3.33% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                                 68.79Ki ± 2%   67.79Ki ±  1%        ~ (p=0.093 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                    76.46Ki ± 3%   67.95Ki ±  2%  -11.12% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                  135.94Ki ± 1%   70.82Ki ±  1%  -47.90% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                             70.93Ki ± 2%   70.50Ki ±  1%        ~ (p=0.284 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                70.99Ki ± 2%   70.92Ki ±  1%        ~ (p=0.799 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                               68.18Ki ± 1%   71.19Ki ±  1%   +4.41% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                           67.54Ki ± 1%   66.95Ki ±  1%        ~ (p=0.180 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                              70.45Ki ± 1%   69.50Ki ±  1%   -1.36% (p=0.009 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                             91.50Ki ± 2%   77.39Ki ±  1%  -15.43% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                          73.78Ki ± 2%   72.58Ki ±  1%        ~ (p=0.065 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                            122.00Ki ± 0%   83.06Ki ±  2%  -31.92% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                            299.5Ki ± 0%   153.2Ki ±  4%  -48.84% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                            70.87Ki ± 2%   70.69Ki ±  2%        ~ (p=0.818 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                               70.33Ki ± 1%   75.17Ki ±  2%   +6.87% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                              68.62Ki ± 2%   89.20Ki ±  1%  +30.00% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                          67.83Ki ± 1%   66.63Ki ±  2%        ~ (p=0.093 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                             70.74Ki ± 1%   67.58Ki ±  1%   -4.47% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                            88.97Ki ± 1%   69.43Ki ±  1%  -21.96% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                         73.74Ki ± 2%   71.68Ki ±  1%   -2.79% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                           114.60Ki ± 1%   71.63Ki ±  1%  -37.49% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                          267.00Ki ± 1%   75.02Ki ±  1%  -71.90% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                               72.24Ki ± 1%   72.37Ki ±  2%        ~ (p=0.699 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                  71.42Ki ± 1%   71.75Ki ±  1%        ~ (p=0.310 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                 69.40Ki ± 1%   70.75Ki ±  1%   +1.95% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                           67.72Ki ± 1%   68.03Ki ±  1%        ~ (p=0.394 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps              68.01Ki ± 1%   67.61Ki ±  1%        ~ (p=0.180 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps             74.88Ki ± 2%   70.06Ki ±  1%   -6.43% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                         70.16Ki ± 2%   70.21Ki ±  2%        ~ (p=0.699 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps            91.64Ki ± 1%   78.77Ki ±  2%  -14.04% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps           156.6Ki ± 1%   142.3Ki ±  4%   -9.14% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                              72.66Ki ± 2%   72.62Ki ±  1%        ~ (p=0.786 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                 72.24Ki ± 2%   73.18Ki ±  1%   +1.31% (p=0.026 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                                69.58Ki ± 1%   74.69Ki ±  2%   +7.34% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                          67.78Ki ± 2%   67.46Ki ±  1%        ~ (p=0.589 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                             68.64Ki ± 2%   67.51Ki ±  1%   -1.65% (p=0.026 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                            75.36Ki ± 2%   67.62Ki ±  1%  -10.26% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                        70.38Ki ± 1%   70.31Ki ±  1%        ~ (p=0.556 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                           92.66Ki ± 1%   70.03Ki ±  1%  -24.43% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                         207.88Ki ± 1%   79.93Ki ±  1%  -61.55% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                  71.46Ki ± 2%   71.30Ki ±  1%        ~ (p=0.513 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                     70.25Ki ± 2%   75.53Ki ±  0%   +7.51% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                    68.05Ki ± 1%   89.29Ki ±  1%  +31.20% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                68.01Ki ± 1%   67.85Ki ±  1%        ~ (p=0.818 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                   69.39Ki ± 3%   67.31Ki ±  2%   -3.00% (p=0.015 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                  87.73Ki ± 1%   69.31Ki ±  2%  -21.00% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                               73.45Ki ± 1%   73.75Ki ±  1%        ~ (p=0.394 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                 115.38Ki ± 1%   71.73Ki ±  1%  -37.83% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                266.31Ki ± 1%   73.55Ki ±  4%  -72.38% (p=0.002 n=6)
geomean                                                                          86.94Ki        80.49Ki         -7.43%

Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes overall look good to me 🚀

Could you please add this PR to the existing changelog entry for the new PromQL engine?

I'm not expecting any dramatic differences, but could you also please compare benchmark results for the non-native histogram selector cases for the new PromQL engine before and after this change? (eg. a_2000,_range_query_with_1000_steps) Just want to make sure there are no unexpected significant regressions.

pkg/streamingpromql/benchmarks/comparison_test.go Outdated Show resolved Hide resolved
pkg/streamingpromql/benchmarks/ingester.go Show resolved Hide resolved
pkg/streamingpromql/benchmarks/ingester.go Show resolved Hide resolved
pkg/streamingpromql/benchmarks/ingester.go Outdated Show resolved Hide resolved
@jhesketh
Copy link
Contributor Author

jhesketh commented May 29, 2024

Benchmarking against main shows no difference within a margin of error:


goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
cpu: 12th Gen Intel(R) Core(TM) i7-1270P
                                                                                     │ benchmark-main.txt │           benchmark-nh.txt           │
                                                                                     │       sec/op       │    sec/op     vs base                │
Query/a_2000,_instant_query/streaming-16                                                     13.61m ± ∞ ¹   18.31m ±  3%       ~ (p=0.286 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                    14.41m ± ∞ ¹   18.54m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                        26.60m ± ∞ ¹   32.01m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                       27.21m ± ∞ ¹   32.44m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                       141.6m ± ∞ ¹   155.4m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                      140.8m ± ∞ ¹   156.2m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                 5.081m ± ∞ ¹   5.729m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                                5.999m ± ∞ ¹   6.755m ±  3%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                            30.45m ± ∞ ¹   32.07m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                           31.80m ± ∞ ¹   33.00m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                               64.36m ± ∞ ¹   56.55m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                              123.1m ± ∞ ¹   106.6m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                              329.8m ± ∞ ¹   276.8m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                            1028.3m ± ∞ ¹   818.0m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                              2.964 ± ∞ ¹    2.675 ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                             9.206 ± ∞ ¹    8.693 ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                                 20.06m ± ∞ ¹   21.53m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                                24.13m ± ∞ ¹   25.30m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                    36.30m ± ∞ ¹   36.64m ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                   75.71m ± ∞ ¹   72.59m ±  4%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                   184.2m ± ∞ ¹   173.2m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                  543.6m ± ∞ ¹   517.3m ±  2%       ~ (p=0.286 n=1+6)

                                                                                     │ benchmark-main.txt │           benchmark-nh.txt            │
                                                                                     │        B/op        │     B/op       vs base                │
Query/a_2000,_instant_query/streaming-16                                                    2.684Mi ± ∞ ¹   2.690Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                   3.019Mi ± ∞ ¹   3.019Mi ±  0%       ~ (p=1.000 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                       3.777Mi ± ∞ ¹   3.771Mi ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                      4.010Mi ± ∞ ¹   3.965Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                      12.26Mi ± ∞ ¹   12.14Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                     12.35Mi ± ∞ ¹   12.34Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                1.921Mi ± ∞ ¹   1.903Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                               2.287Mi ± ∞ ¹   2.291Mi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                           5.942Mi ± ∞ ¹   5.940Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                          8.178Mi ± ∞ ¹   8.178Mi ±  0%       ~ (p=1.000 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                              8.071Mi ± ∞ ¹   8.094Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                             10.16Mi ± ∞ ¹   10.22Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                             24.85Mi ± ∞ ¹   24.82Mi ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                           121.77Mi ± ∞ ¹   27.94Mi ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                            701.8Mi ± ∞ ¹   700.4Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                           3.036Gi ± ∞ ¹   3.035Gi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                                3.937Mi ± ∞ ¹   3.930Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                               6.164Mi ± ∞ ¹   6.160Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                   5.301Mi ± ∞ ¹   5.339Mi ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                  7.707Mi ± ∞ ¹   7.703Mi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                  16.38Mi ± ∞ ¹   16.40Mi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                 20.78Mi ± ∞ ¹   20.74Mi ±  1%       ~ (p=0.571 n=1+6)


                                                                                     │ benchmark-main.txt │          benchmark-nh.txt           │
                                                                                     │     allocs/op      │  allocs/op   vs base                │
Query/a_2000,_instant_query/streaming-16                                                     38.80k ± ∞ ¹   38.80k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                    38.87k ± ∞ ¹   38.89k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                        46.85k ± ∞ ¹   46.84k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                       46.97k ± ∞ ¹   46.92k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                       110.9k ± ∞ ¹   110.9k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                      111.0k ± ∞ ¹   110.9k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                 30.75k ± ∞ ¹   30.76k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                                30.84k ± ∞ ¹   30.85k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                            93.63k ± ∞ ¹   93.61k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                           89.79k ± ∞ ¹   89.80k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                               109.7k ± ∞ ¹   109.7k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                              102.1k ± ∞ ¹   102.1k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                              237.8k ± ∞ ¹   237.8k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                             245.9k ± ∞ ¹   232.0k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                             1.446M ± ∞ ¹   1.446M ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                            1.485M ± ∞ ¹   1.484M ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                                 58.17k ± ∞ ¹   58.17k ± 0%       ~ (p=1.000 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                                66.59k ± ∞ ¹   66.59k ± 0%       ~ (p=1.000 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                    67.34k ± ∞ ¹   67.35k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                   75.81k ± ∞ ¹   75.81k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                   136.5k ± ∞ ¹   136.4k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                  173.7k ± ∞ ¹   173.7k ± 0%       ~ (p=1.000 n=1+6)

                                                                                     │ benchmark-main.txt │           benchmark-nh.txt            │
                                                                                     │         B          │       B        vs base                │
Query/a_2000,_instant_query/streaming-16                                                    68.55Ki ± ∞ ¹   68.36Ki ±  1%       ~ (p=0.857 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                   68.18Ki ± ∞ ¹   68.68Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                       77.14Ki ± ∞ ¹   76.92Ki ±  1%       ~ (p=0.857 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                      75.49Ki ± ∞ ¹   75.65Ki ±  2%       ~ (p=0.857 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                      136.2Ki ± ∞ ¹   136.0Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                     134.5Ki ± ∞ ¹   135.6Ki ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                70.96Ki ± ∞ ¹   70.34Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                               70.71Ki ± ∞ ¹   70.67Ki ±  1%       ~ (p=1.000 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                           69.43Ki ± ∞ ¹   71.12Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                          69.84Ki ± ∞ ¹   70.00Ki ±  1%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                              78.62Ki ± ∞ ¹   80.00Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                             92.00Ki ± ∞ ¹   93.87Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                             142.0Ki ± ∞ ¹   142.2Ki ±  2%       ~ (p=1.000 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                            156.4Ki ± ∞ ¹   207.9Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                            469.2Ki ± ∞ ¹   459.0Ki ±  3%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                           1.564Mi ± ∞ ¹   1.578Mi ±  2%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                               102.75Ki ± ∞ ¹   97.06Ki ±  3%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                               68.36Ki ± ∞ ¹   70.06Ki ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                  132.22Ki ± ∞ ¹   99.19Ki ± 32%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                  83.38Ki ± ∞ ¹   84.11Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                  123.1Ki ± ∞ ¹   124.4Ki ±  8%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                 161.2Ki ± ∞ ¹   160.4Ki ±  0%       ~ (p=0.286 n=1+6)

Copy link
Contributor

@charleskorn charleskorn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@jhesketh jhesketh merged commit 6db3385 into grafana:main May 29, 2024
29 checks passed
narqo pushed a commit to narqo/grafana-mimir that referenced this pull request Jun 6, 2024
* Streaming PromQL engine: native histograms

Add native histogram support to instant vectors.

* Use seriesPerResultBucketFactor for histogram pool

* Don't defer returning slices

* Refactor slice returning logic

* Move returnSlices helper

* rename val for consistency

* Update comment from feedback

* Use returnSeriesDataSlices where missed

* Comment out unsupported tests for now

* Enable upstream native_histogram test file

* Remove duplicate test

* Fix FloatHistogram selector

* Add range test

* Add test for mixed floats+histograms

* Re-enable supported upstream tests

* Load benchmark samples in batches to avoid OOM'ing

* Revert "Load benchmark samples in batches to avoid OOM'ing"

We can't use OOO samples with native histograms, so instead of
batching by series we'll batch by samples

This reverts commit 91ba5c1.

* Batch by samples instead of series

* Generate native histogram benchmark test data

* Add native histogram benchmark test

* Move compare.sh next to other benchmark scripts

* Fix comparison tests for when there are no floats

* Add extra samples to the histogram tests

* Add note on batch size

* Add quick check that histograms are loaded

* Use clearer variables for indexes

* Update changelog

* Fix typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants