Mimir query engine: add support for `min_over_time`, `max_over_time`, `sum_over_time` and `avg_over_time` #8934

charleskorn · 2024-08-08T05:53:24Z

What this PR does

This PR adds support for the following functions to MQE:

min_over_time
max_over_time
sum_over_time
avg_over_time

tl;dr of benchmark results: sum_over_time and avg_over_time are up to 30% faster for floats and up to 50% faster for native histograms compared to Prometheus' engine. Peak memory consumption is largely the same, which is unsurprising given the final result dominates memory consumption in the benchmark cases.

Benchmark results for sum_over_time and avg_over_time, compared to Prometheus' engine

goos: darwin
goarch: arm64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
                                                              │      Prometheus      │                Mimir                │
                                                              │        sec/op        │    sec/op     vs base               │
Query/avg_over_time(a_1[1m]),_instant_query                              139.7µ ± 4%   142.8µ ±  4%        ~ (p=0.093 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_100_steps                 162.8µ ± 2%   150.0µ ±  3%   -7.88% (p=0.002 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_1000_steps                287.6µ ± 4%   255.8µ ±  2%  -11.06% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_instant_query                            496.1µ ± 2%   396.0µ ±  2%  -20.18% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_100_steps               1.688m ± 2%   1.300m ±  4%  -23.00% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_1000_steps             12.202m ± 3%   8.835m ±  4%  -27.59% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_instant_query                           5.216m ± 4%   4.199m ±  4%  -19.49% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_100_steps              27.74m ± 2%   20.48m ±  1%  -26.16% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_1000_steps             221.1m ± 2%   161.0m ± 12%  -27.21% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_instant_query                             141.1µ ± 3%   134.1µ ±  3%   -4.94% (p=0.009 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_100_steps                483.6µ ± 6%   316.4µ ±  4%  -34.57% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_1000_steps               3.157m ± 1%   1.690m ±  3%  -46.46% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_instant_query                           884.4µ ± 9%   656.2µ ±  1%  -25.81% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_100_steps              29.57m ± 1%   15.58m ±  1%  -47.31% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_1000_steps             283.6m ± 1%   144.2m ±  1%  -49.17% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_instant_query                         12.265m ± 1%   8.110m ±  1%  -33.88% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_100_steps             571.1m ± 2%   289.8m ±  1%  -49.25% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_1000_steps             5.490 ± 1%    2.890 ±  1%  -47.36% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_instant_query                              135.2µ ± 3%   129.3µ ±  7%        ~ (p=0.132 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_100_steps                 147.5µ ± 2%   141.2µ ±  6%        ~ (p=0.065 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_1000_steps                266.2µ ± 1%   227.1µ ±  1%  -14.69% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_instant_query                            479.9µ ± 3%   372.1µ ±  1%  -22.48% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_100_steps               1.596m ± 1%   1.198m ±  0%  -24.89% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_1000_steps             11.277m ± 1%   8.013m ±  1%  -28.94% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_instant_query                           5.201m ± 2%   4.084m ±  2%  -21.48% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_100_steps              25.76m ± 1%   18.35m ±  1%  -28.79% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_1000_steps             209.3m ± 1%   144.8m ±  1%  -30.80% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_instant_query                             133.7µ ± 2%   131.5µ ±  1%   -1.65% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_100_steps                237.5µ ± 1%   217.3µ ±  1%   -8.53% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_1000_steps              1006.6µ ± 5%   882.5µ ±  1%  -12.32% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_instant_query                           675.0µ ± 2%   571.2µ ±  2%  -15.38% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_100_steps              8.401m ± 3%   7.198m ±  0%  -14.32% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_1000_steps             74.66m ± 1%   63.90m ±  0%  -14.41% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_instant_query                          8.491m ± 1%   6.586m ±  2%  -22.43% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_100_steps             154.2m ± 0%   132.3m ±  1%  -14.20% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_1000_steps             1.486 ± 1%    1.270 ±  1%  -14.54% (p=0.002 n=6)
geomean                                                                  4.838m        3.702m        -23.47%

                                                              │      Prometheus      │                Mimir                │
                                                              │         B/op         │     B/op      vs base               │
Query/avg_over_time(a_1[1m]),_instant_query                             23.82Ki ± 0%   21.97Ki ± 0%   -7.75% (p=0.002 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_100_steps                24.63Ki ± 0%   22.42Ki ± 0%   -8.98% (p=0.002 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_1000_steps               29.63Ki ± 0%   27.32Ki ± 0%   -7.78% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_instant_query                           135.3Ki ± 0%   118.6Ki ± 0%  -12.30% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_100_steps              175.3Ki ± 0%   163.0Ki ± 0%   -7.00% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_1000_steps             651.0Ki ± 0%   639.0Ki ± 0%   -1.84% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_instant_query                          2.249Mi ± 0%   1.917Mi ± 0%  -14.74% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_100_steps             3.018Mi ± 0%   2.840Mi ± 0%   -5.88% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_1000_steps            12.11Mi ± 0%   11.94Mi ± 1%   -1.39% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_instant_query                            32.85Ki ± 0%   28.71Ki ± 0%  -12.62% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_100_steps               370.9Ki ± 0%   111.0Ki ± 0%  -70.09% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_1000_steps             3332.5Ki ± 0%   798.8Ki ± 0%  -76.03% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_instant_query                          455.0Ki ± 0%   208.2Ki ± 0%  -54.25% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_100_steps            32.545Mi ± 0%   7.639Mi ± 0%  -76.53% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_1000_steps           321.65Mi ± 0%   74.92Mi ± 0%  -76.71% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_instant_query                         8.356Mi ± 0%   3.555Mi ± 0%  -57.46% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_100_steps            650.7Mi ± 0%   153.5Mi ± 0%  -76.41% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_1000_steps           6.281Gi ± 0%   1.482Gi ± 0%  -76.41% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_instant_query                             23.76Ki ± 0%   21.99Ki ± 0%   -7.47% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_100_steps                24.61Ki ± 1%   22.42Ki ± 0%   -8.92% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_1000_steps               29.60Ki ± 0%   27.29Ki ± 0%   -7.82% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_instant_query                           135.2Ki ± 0%   118.6Ki ± 0%  -12.23% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_100_steps              175.4Ki ± 0%   162.7Ki ± 0%   -7.24% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_1000_steps             650.2Ki ± 0%   637.8Ki ± 0%   -1.92% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_instant_query                          2.248Mi ± 0%   1.919Mi ± 0%  -14.66% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_100_steps             3.015Mi ± 0%   2.835Mi ± 0%   -5.98% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_1000_steps            12.11Mi ± 0%   11.97Mi ± 0%   -1.17% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_instant_query                            30.30Ki ± 0%   28.22Ki ± 0%   -6.87% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_100_steps               67.11Ki ± 0%   60.02Ki ± 0%  -10.58% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_1000_steps              314.6Ki ± 0%   293.8Ki ± 0%   -6.63% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_instant_query                          202.7Ki ± 0%   157.8Ki ± 0%  -22.16% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_100_steps             2.871Mi ± 0%   2.683Mi ± 0%   -6.58% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_1000_steps            26.81Mi ± 0%   25.25Mi ± 0%   -5.80% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_instant_query                         3.429Mi ± 0%   2.555Mi ± 0%  -25.50% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_100_steps            56.39Mi ± 0%   52.60Mi ± 0%   -6.71% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_1000_steps           566.7Mi ± 0%   539.6Mi ± 0%   -4.79% (p=0.002 n=6)
geomean                                                                 1.239Mi        892.2Ki       -29.66%

                                                              │      Prometheus      │               Mimir                │
                                                              │      allocs/op       │  allocs/op   vs base               │
Query/avg_over_time(a_1[1m]),_instant_query                               430.0 ± 0%    391.0 ± 0%   -9.07% (p=0.002 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_100_steps                  440.0 ± 0%    396.0 ± 0%  -10.00% (p=0.002 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_1000_steps                 473.0 ± 0%    429.0 ± 0%   -9.30% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_instant_query                            1.954k ± 0%   1.900k ± 0%   -2.76% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_100_steps               2.364k ± 0%   2.307k ± 0%   -2.43% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_1000_steps              5.680k ± 0%   5.623k ± 0%   -1.00% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_instant_query                           30.87k ± 0%   30.79k ± 0%   -0.25% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_100_steps              38.96k ± 0%   38.90k ± 0%   -0.14% (p=0.002 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_1000_steps             105.0k ± 0%   105.0k ± 0%   -0.05% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_instant_query                              559.0 ± 0%    473.0 ± 0%  -15.38% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_100_steps                7.361k ± 0%   2.103k ± 0%  -71.43% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_1000_steps               66.87k ± 0%   15.69k ± 0%  -76.54% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_instant_query                           8.532k ± 0%   3.864k ± 0%  -54.71% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_100_steps              669.8k ± 0%   154.9k ± 0%  -76.88% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_1000_steps             6.618M ± 0%   1.512M ± 0%  -77.15% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_instant_query                         161.04k ± 0%   68.87k ± 0%  -57.23% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_100_steps            13.382M ± 0%   3.088M ± 0%  -76.92% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_1000_steps           132.33M ± 0%   30.24M ± 0%  -77.15% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_instant_query                               430.0 ± 0%    391.0 ± 0%   -9.07% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_100_steps                  440.0 ± 0%    396.0 ± 0%  -10.00% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_1000_steps                 473.0 ± 0%    429.0 ± 0%   -9.30% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_instant_query                            1.954k ± 0%   1.900k ± 0%   -2.76% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_100_steps               2.364k ± 0%   2.306k ± 0%   -2.45% (p=0.002 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_1000_steps              5.679k ± 0%   5.622k ± 0%   -1.01% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_instant_query                           30.87k ± 0%   30.79k ± 0%   -0.25% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_100_steps              38.96k ± 0%   38.89k ± 0%   -0.17% (p=0.002 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_1000_steps             105.0k ± 0%   105.0k ± 0%   -0.05% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_instant_query                              509.0 ± 0%    463.0 ± 0%   -9.04% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_100_steps                1.306k ± 0%   1.092k ± 0%  -16.39% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_1000_steps               6.785k ± 0%   5.671k ± 0%  -16.42% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_instant_query                           3.526k ± 0%   2.863k ± 0%  -18.80% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_100_steps              64.56k ± 0%   53.83k ± 0%  -16.62% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_1000_steps             611.9k ± 0%   511.2k ± 0%  -16.46% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_instant_query                          60.97k ± 0%   48.86k ± 0%  -19.86% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_100_steps             1.277M ± 0%   1.065M ± 0%  -16.61% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_1000_steps            12.23M ± 0%   10.22M ± 0%  -16.45% (p=0.002 n=6)
geomean                                                                  20.46k        14.37k       -29.79%

                                                              │      Prometheus      │                Mimir                │
                                                              │          B           │      B        vs base               │
Query/avg_over_time(a_1[1m]),_instant_query                             71.51Mi ± 1%   71.27Mi ± 1%        ~ (p=0.420 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_100_steps                71.37Mi ± 2%   71.22Mi ± 2%        ~ (p=0.788 n=6)
Query/avg_over_time(a_1[1m]),_range_query_with_1000_steps               70.10Mi ± 1%   69.16Mi ± 1%   -1.35% (p=0.002 n=6)
Query/avg_over_time(a_100[1m]),_instant_query                           66.16Mi ± 2%   65.92Mi ± 1%        ~ (p=0.167 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_100_steps              65.81Mi ± 1%   65.36Mi ± 1%        ~ (p=0.234 n=6)
Query/avg_over_time(a_100[1m]),_range_query_with_1000_steps             66.55Mi ± 3%   67.61Mi ± 1%        ~ (p=0.180 n=6)
Query/avg_over_time(a_2000[1m]),_instant_query                          68.28Mi ± 1%   68.25Mi ± 1%        ~ (p=0.699 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_100_steps             74.06Mi ± 2%   74.52Mi ± 1%        ~ (p=0.240 n=6)
Query/avg_over_time(a_2000[1m]),_range_query_with_1000_steps            127.1Mi ± 1%   131.3Mi ± 1%   +3.31% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_instant_query                            77.48Mi ± 1%   78.83Mi ± 2%   +1.74% (p=0.026 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_100_steps               68.59Mi ± 1%   70.41Mi ± 1%   +2.64% (p=0.002 n=6)
Query/avg_over_time(nh_1[1m]),_range_query_with_1000_steps              67.95Mi ± 1%   68.80Mi ± 0%   +1.26% (p=0.002 n=6)
Query/avg_over_time(nh_100[1m]),_instant_query                          68.37Mi ± 1%   67.29Mi ± 2%   -1.58% (p=0.026 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_100_steps             69.54Mi ± 1%   70.67Mi ± 1%   +1.63% (p=0.009 n=6)
Query/avg_over_time(nh_100[1m]),_range_query_with_1000_steps            116.0Mi ± 1%   118.2Mi ± 1%   +1.86% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_instant_query                         69.86Mi ± 2%   71.43Mi ± 2%   +2.25% (p=0.017 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_100_steps            168.6Mi ± 2%   176.0Mi ± 1%   +4.41% (p=0.002 n=6)
Query/avg_over_time(nh_2000[1m]),_range_query_with_1000_steps          1012.8Mi ± 5%   810.6Mi ± 7%  -19.97% (p=0.002 n=6)
Query/sum_over_time(a_1[1m]),_instant_query                             72.29Mi ± 1%   71.55Mi ± 2%   -1.02% (p=0.006 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_100_steps                71.87Mi ± 1%   71.27Mi ± 1%        ~ (p=0.143 n=6)
Query/sum_over_time(a_1[1m]),_range_query_with_1000_steps               70.18Mi ± 1%   69.41Mi ± 1%        ~ (p=0.065 n=6)
Query/sum_over_time(a_100[1m]),_instant_query                           66.32Mi ± 1%   66.00Mi ± 1%        ~ (p=0.102 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_100_steps              65.41Mi ± 1%   65.33Mi ± 1%        ~ (p=0.907 n=6)
Query/sum_over_time(a_100[1m]),_range_query_with_1000_steps             66.45Mi ± 3%   67.64Mi ± 1%        ~ (p=0.071 n=6)
Query/sum_over_time(a_2000[1m]),_instant_query                          68.23Mi ± 1%   67.93Mi ± 1%        ~ (p=0.240 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_100_steps             73.11Mi ± 2%   74.96Mi ± 1%   +2.53% (p=0.015 n=6)
Query/sum_over_time(a_2000[1m]),_range_query_with_1000_steps            126.8Mi ± 1%   131.4Mi ± 1%   +3.59% (p=0.002 n=6)
Query/sum_over_time(nh_1[1m]),_instant_query                            79.18Mi ± 2%   78.98Mi ± 1%        ~ (p=0.420 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_100_steps               74.46Mi ± 1%   74.43Mi ± 1%        ~ (p=0.818 n=6)
Query/sum_over_time(nh_1[1m]),_range_query_with_1000_steps              71.09Mi ± 1%   70.02Mi ± 1%   -1.51% (p=0.026 n=6)
Query/sum_over_time(nh_100[1m]),_instant_query                          71.70Mi ± 1%   68.15Mi ± 1%   -4.95% (p=0.002 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_100_steps             71.16Mi ± 1%   71.72Mi ± 1%        ~ (p=0.132 n=6)
Query/sum_over_time(nh_100[1m]),_range_query_with_1000_steps            119.6Mi ± 1%   120.0Mi ± 1%        ~ (p=0.372 n=6)
Query/sum_over_time(nh_2000[1m]),_instant_query                         74.48Mi ± 1%   71.52Mi ± 2%   -3.98% (p=0.002 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_100_steps            176.0Mi ± 1%   177.0Mi ± 1%        ~ (p=0.071 n=6)
Query/sum_over_time(nh_2000[1m]),_range_query_with_1000_steps           612.1Mi ± 2%   603.2Mi ± 0%   -1.46% (p=0.037 n=6)
geomean                                                                 89.90Mi        89.58Mi        -0.36%

Which issue(s) this PR fixes or relates to

(none)

Checklist

Tests updated.
[n/a] Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
[n/a] about-versioning.md updated with experimental features.

…time` and `present_over_time`

…native histograms

charleskorn · 2024-08-08T22:13:00Z

Putting this back to draft to incorporate changes from prometheus/prometheus#14413

jhesketh

I know it's a draft. Just getting a head start :-)

jhesketh · 2024-08-09T06:58:24Z

pkg/streamingpromql/engine_test.go

@@ -1405,6 +1405,46 @@ func TestAnnotations(t *testing.T) {
 			},
 		},

+		"sum_over_time() over series with both floats and histograms": {


(nit)
It might be worth splitting out the _over_time functions into their own test so we can check the common case of mixed metrics etc iterating over the function.

I'm inclined to leave this as-is for the time being - until prometheus/prometheus#14609 is vendored into Mimir, we need to skip one of the rate test cases that would fall into this group of tests, and I don't think it's so bad that the tests are repeated.

pkg/streamingpromql/functions/range_vectors.go

pkg/streamingpromql/testdata/ours/functions.test

# Conflicts: # pkg/streamingpromql/testdata/upstream/functions.test

…from prometheus/prometheus#14413

charleskorn force-pushed the charleskorn/more-over-time-functions branch 2 times, most recently from 0eaf8d9 to 33fef8f Compare August 8, 2024 07:22

charleskorn added 7 commits August 8, 2024 17:25

Ensure non-name labels are retained in count_over_time, `last_over_…

2ce5275

…time` and `present_over_time`

Enable test cases for min_over_time and max_over_time

294c807

Add min_over_time and max_over_time

68408e7

Add support for sum_over_time

07777fc

Add support for avg_over_time

28ff94c

Expand concurrent queries test to include new functions that support …

5bf52b4

…native histograms

Add changelog entry

dc3b6a9

charleskorn force-pushed the charleskorn/more-over-time-functions branch from 1794e9e to dc3b6a9 Compare August 8, 2024 07:26

charleskorn changed the title ~~Mimir query engine: add support for min_over_time and max_over_time~~ Mimir query engine: add support for min_over_time, max_over_time, sum_over_time and avg_over_time Aug 8, 2024

charleskorn marked this pull request as ready for review August 8, 2024 07:29

charleskorn requested a review from a team as a code owner August 8, 2024 07:29

charleskorn marked this pull request as draft August 8, 2024 22:13

Fix typo in test

20cc9cf

jhesketh reviewed Aug 9, 2024

View reviewed changes

charleskorn added 6 commits August 9, 2024 17:28

Don't create unnecessary FloatHistogram instances in avg_over_time

a8e8b0e

Add benchmarks

82fe467

Add test cases for combinations of infinity and NaN

4de9410

Explain why we need to copy histograms in avg_over_time

ae939cb

Merge branch 'main' into charleskorn/more-over-time-functions

b921558

# Conflicts: # pkg/streamingpromql/testdata/upstream/functions.test

Use simpler and more accurate average calculation in avg_over_time …

5702f1a

…from prometheus/prometheus#14413

charleskorn marked this pull request as ready for review August 12, 2024 01:47

jhesketh approved these changes Aug 12, 2024

View reviewed changes

charleskorn merged commit 156c1f9 into main Aug 12, 2024
29 checks passed

charleskorn deleted the charleskorn/more-over-time-functions branch August 12, 2024 04:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mimir query engine: add support for `min_over_time`, `max_over_time`, `sum_over_time` and `avg_over_time` #8934

Mimir query engine: add support for `min_over_time`, `max_over_time`, `sum_over_time` and `avg_over_time` #8934

charleskorn commented Aug 8, 2024 •

edited

Loading

charleskorn commented Aug 8, 2024

jhesketh left a comment

jhesketh Aug 9, 2024

charleskorn Aug 12, 2024

Mimir query engine: add support for min_over_time, max_over_time, sum_over_time and avg_over_time #8934

Mimir query engine: add support for min_over_time, max_over_time, sum_over_time and avg_over_time #8934

Conversation

charleskorn commented Aug 8, 2024 • edited Loading

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

charleskorn commented Aug 8, 2024

jhesketh left a comment

Choose a reason for hiding this comment

jhesketh Aug 9, 2024

Choose a reason for hiding this comment

charleskorn Aug 12, 2024

Choose a reason for hiding this comment

Mimir query engine: add support for `min_over_time`, `max_over_time`, `sum_over_time` and `avg_over_time` #8934

Mimir query engine: add support for `min_over_time`, `max_over_time`, `sum_over_time` and `avg_over_time` #8934

charleskorn commented Aug 8, 2024 •

edited

Loading