Ability to use multiple dedup labels

[jpiper]

Let's say we have two external services

Service A
Service B

Which are scraped by two prometheus setups in two kubernetes clusters (so there's two prometheus instances in each site)

site X: P1 and P2
site Y: P1 and P2

And then we have a "global" thanos querier that uses the store API of the thanos query nodes in both of these sites.

So we end up with metrics like

up{service="A",site="X",prometheus_replica="1"}
up{service="A",site="X",prometheus_replica="2"}
up{service="A",site="Y",prometheus_replica="1"}
up{service="A",site="Y",prometheus_replica="2"}
up{service="B",site="X",prometheus_replica="1"}
up{service="B",site="X",prometheus_replica="2"}
up{service="B",site="Y",prometheus_replica="1"}
up{service="B",site="Y",prometheus_replica="2"}

but we really just want

up{service="A"}
up{service="B"}

Where we want to dedup across site and prometheus_replica. What is the best approach here? Can the global thanos querier ask the site X and site Y to dedup, and then the global just dedups on site? Or is it possible to provide multiple dedup labels?

[bwplotka]

Hey, thanks for this (:

The current way is just use single label. In your case, literally remove prometheus_replica and just use site (: OR remove site and add replica=siteX (:

We could add multi label, but what's the reason for that if you remove thie extra metadata immdiately on Thanos Query API? so why do you need this multiple labels info? If you will never filter on that?

The only use case would be if you sometimes deduplicate and for other system you don't deduplicate, is that a case? (:

[metalmatze]

I'm currently working on a setup that involves kube-prometheus and kube-thanos.

Let's use node_boot_time_seconds as example.
We have 2 replicas of Prometheus running and collecting that metric from node-exporter. Next to that we have Thanos running queriers, stores and receive components. Once the Prometheus ingest a time series, they remote write those to the Thanos receive component, adding prometheus_replica as label.
Once we query the Thanos queriers they talk to the receive components and those reply with their additional replica label. Now we end up with both replica and prometheus_replica as labels we want to dedup.

/cc @brancz @squat

[bwplotka]

So essentially this is 4 replications of same data? I would say maybe we need some deduplication during writes, but I think that use case for time being makes sense, we should do it.

[brancz]

I think currently this is not a problem yet, as we don't actually replicate any data, so deduplication is still just the prometheus_replica label. While replica from the Thanos receive component may be there currently, it doesn't actually require a dedup, as it's orthogonal to the deduplication. What we do need is that the label(s) to dedup should be configurable at request time, as with the receive component we can't really make a global assumption (thinking further, I'm not even sure we can currently make a global assumption, instead we're currently forcing this on users). The current mode of enforcing the dedup label globally has worked out surprisingly well so I'm not inclined to remove it, but instead add a flag that allows them to be passed as part of the query request, so a user of the receive component can decide on a per "tenant" basis.

[squat]

Now that replication by the receive component has merged (#1270), this is more pressing. I’ll do some digging :)

[bwplotka]

AC:

• Allow deduplication with multiple labels instead of just one (e.g use repeated replica-label flag)
• Allow specifying deduplication label on run time via extra parameter to query and query_range
• Document changes

cc @krasi-georgiev

[krasi-georgiev]

Thanks, looking into it.

[krasi-georgiev]

Allow specifying deduplication label on run time via extra parameter to query and query_range

I imagine the query parameter would override the fixed replica-label flags ?

[jpiper]

Thanks for the enhancement, this is perfect!