promxy in front of thanos query-frontend

[mushrushu]

Hello!

I'm having issues using promxy in front of thanos query frontend.

We have a prometheus instance with fairly recent metrics, there is also a thanos instance provided by external team for long term metrics storage. We decided to use promxy as a glue for both instances to have a unified metrics datasource in Grafana.

I've set this configuration up but saw and error in promxy webui: "Warning: Error fetching metrics list: Unexpected response status when fetching metric names: Unprocessable Entity". I've tested configurations with single datasource and the error persisted with thanos only.

As far as I can tell, promxy tries a get request to /api/v1/label/name/values with parameters end=9223309901257974&start=-9223309901257974 to thanos and it gets "end timestamp must not be before start time" in return. However prometheus is doing just fine in such a scenario.

my promxy config is:

promxy:
  server_groups:
    - static_configs:
        - targets:
            - "thanos:9090"

Am I missing something in my config file? Is it even a valid scenario for promxy?

[jacksontj]

I'm actually a bit surprised you would see this on latest as the time shouldn't be sent down (fixed in #575 ). If you are still seeing this we'll have to dig in some why. I imagine this is an interplay between the prometheus API and Thanos though. prometheus (for some reason) has mintime as that massive negative number -- which some systems don't handle. Technically prometheus APIs should support that time range (even though it is a bit crazy); but as a workaround you could use an absolute time range with truncate to make sure the start time isn't so negative.

#575

[mushrushu]

My bad, originally I was testing on v0.0.77, so the described behaviour was for that version. However, when a new version became available [v0.0.78] I tried it but was not thorough enough. I saw the same behaviour in ui and didn't do anything else.
Now I see that there is now start parameter but the end parameter is still there with the same value as before and the result is still the same. "end timestamp must not be before start time"

Do I understand correctly that with the current version both time parameters (i.e. start/end) should be gone?

Should relative_time_range have the same effect for the matter as absolute_time_range?

Will try absolute_time_range anyway and report back later. Thanks.

[mushrushu]

I've tried to use absolute time range but to no avail. There request parameters were still the same (i.e. end=9223309901257974) and I've still got the same error "end timestamp must not be before start time".

my promxy config was:

promxy:
  server_groups:
    - static_configs:
        - targets:
            - "thanos:9090"
      absolute_time_range:
        start: '2023-03-01T01:00:00Z'
        end: '2023-03-26T23:00:00Z'
        truncate: true 

[jacksontj]

I got some time today to dig into this -- and congrats you have found a prometheus quirk! Although prometheus has a defined min and max time which include nanosecond precision -- it effectively truncates down to a millisecond timestamp before passing down to the querier (surprisingly it rounds instead of truncates, but just odd -- prometheus/prometheus#4939).

So to fix this we can simply truncate the min/max time to millisecond precision and it works.

In addition, you uncovered a bug in the time filter logic -- it doesn't filter time for LabelNames or LabelValues (the filter feature was added prior to those APIs being exposed). I'll also have a fix for that as well!