Working CLI help / example for --proxy-headers

[urgerestraint]

Hi, I have been validating promxy for merging multi-tenant Mimir query-frontends, which requires setting a X-Scope-OrgID tenant request header on all query/metadata/labels requests.

My read of the ProxyHeaders config option seems to suit this but I was unable to find a working way of supplying a header key value pair with either --proxy-headers or the PROXY_HEADERS envvar. My understanding is the supplied value should be unmarshalled to a []string, but it's not clear on the expected format in the help command. Running promxy in debug and trace log level doesn't appear to show any headers being added to requests to the configured server groups.

Able to hack around this with a reverse proxy setting the header between promxy and the query API, and force setting a header there, but would you be able to provide an example of a working set of headers using --proxy-headers or PROXY_HEADERS? Is there any equivalent way to set this in the config file instead?

Thanks 🙏🏻

[jacksontj]

but I was unable to find a working way of supplying a header key value pair

It seems that maybe the description/help for proxy-headers unfortunately wasn't clear enough :( proxy-headers is a list of header keys that will be proxied from the client to the downstream. This was specifically done for use-cases where the client (browser in grafana) has basic auth that we don't want to configure promxy with (#322).

Your use-case sounds like you want to simply add fixed headers to the downstream requests -- which unfortunately upstream prometheus doesn't support (it has been a topic of discussion in the distant past -- prometheus/prometheus#1724).

From a quick look this should actually be easy to add into promxy as we're already wrapping the prometheus config today.

So if I'm understanding your request correctly; this would actually change the issue here to a feature request of "Add ability to set static headers for requests to servergroups" -- does that sound right?

[urgerestraint]

So if I'm understanding your request correctly; this would actually change the issue here to a feature request of "Add ability to set static headers for requests to servergroups" -- does that sound right?

Yes that appears to be the functionality I'm after, given my misunderstanding of the proxy-headers flag (apologies).

An "Add ability to set static headers for requests to servergroups" feature would allow us to remove intermediate reverse proxies between promxy and prom clusters that are used only to add in fixed headers per servergroup.

This only (currently) appears to be useful to support Mimir and Cortex Multitenant query APIs as they both require X-Scope-OrgID headers in order to perform any query, even if it's the default tenant. https://cortexmetrics.io/docs/guides/auth/

[urgerestraint]

@jacksontj I've added #646 as an example which I've confirmed working with Mimir, keen for a review.