[thanos] Changes to address querying multi cluster reads using thanos query

[rishabhkumar92]

Changes

This change is meant to address querying multi cluster reads using thanos query. I have been trying to make it work with Grafana Mimir backend(Thanos querier -> Thanos sidecar -> Grafana Mimir) which expects X-Scope-OrgID as header for all prometheus APIs. It addresses following fixes:

• Fixes around way we deals with limit options for metadata API (series, label etc), due to limit being set to 0, metadata APIs were not working (ref)
• Set tenant header while querying the underlying backend(Mimir in this example) for thanos query and sidecar. Querier needs to set it while issuing grpc request to sidecar and sidecar needs to expect tenant-flag (X-Scope-OrgID in this case) and pass it to underlying backend (Grafana Mimir in this case)
• Change testcases accordingly

Verification

• Tested locally for different APIs over Grafana Mimir

cc: @aallawala @callumj @edma2 @GiedriusS

[saswatamcode]

@rishabhkumar92 could you specify what use case this change fulfils? Not quite sure what you are trying to achieve with this

[rishabhkumar92]

@rishabhkumar92 could you specify what use case this change fulfils? Not quite sure what you are trying to achieve with this

@saswatamcode I have been trying to use thanos querier + thanos sidecar to query Grafana Mimir which expects X-Scope-OrgID header to set while issuing any APIs (remote-read or prometheus http V1 api) so one set of changes are related to that (passing tenant header from querier to sidecar and sidecar setting it while issuing query to prometheus backend).
Other are related to metadata querying and deal with limits only if it's > 0 due to this. I am also updating description accordingly.

[GiedriusS]

Thanks for your contribution. Why not use something like https://github.com/thanos-community/thanos-promql-connector?

[rishabhkumar92]

Thanks for your contribution. Why not use something like https://github.com/thanos-community/thanos-promql-connector?

Thanks for suggestion. Using promql-connector will require similar changes like i added in this change but in 2 projects(both thanos and connector) because querier side changes can't be avoided.
Let me know what you think of current change and i can change the name(tenant-header) to something else if thats a concern.

[rishabhkumar92]

@GiedriusS can you help review this change.