Sidecar: use prometheus metrics for min timestamp #7820
Conversation
Read "minT" from prometheus metrics so that we also set it for sidecars that are not uploading blocks. Signed-off-by: Michael Hoffmann <mhoffm@posteo.de>
MichaHoffmann
force-pushed
the
mhoffmann/use-prometheus-tsdb-lowest-timestamp-for-mintime-in-sidecar
branch
from
51c58f7
to
6cff6e2
Compare
| // and extracting the prometheus_tsdb_lowest_timestamp_seconds metric from it. | ||
| func (c *Client) LowestTimestamp(ctx context.Context, base *url.URL) (int64, error) { | ||
| u := *base | ||
| u.Path = path.Join(u.Path, "/metrics") |
There was a problem hiding this comment.
Well, it sucks we can't get this from API. I was wondring if we could get those maybe with min(min_over_time(timestamp({__name__!=""}[30d]))), but I guess we would need to know the retention before hand (and it would't be cheap for Prometheus).
There was a problem hiding this comment.
Yeah the approach to read it from /metrics is a bit janky, but i cannot think of another one. That query would probably be prohibitively expensive
It doesn't have this data for whole tsdb. Only for head |
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if mint < s.limitMinTime.PrometheusTimestamp() { |
There was a problem hiding this comment.
While going through this function we can probably update to use the min() builtin
There was a problem hiding this comment.
Oh yeah, good point; ill update in a bit!
This PR makes sidecar read "minT" from prometheus metrics instead of the shipper. This has the benefit that it also works for sidecar deployments that do not upload blocks, so that they can be pruned by time range too.