query-frontend: Split and cache metadata API responses by time interval

[kakkoyun]

We recently added time range parameters to the LabelNames, LabelValues and Series API endpoints for both Thanos Query API.
Now we expect those endpoints to be cached by Query Frontend to increase query scalability. In order to achieve this goal we need to introduce splitting request by time ranges.

Tasks

• Introduce Split by time interval for Label (LabelNames and Label Values) API
• Cache Label API responses
• Introduce Split by time interval for Series API
• Cache Series API response

Help wanted :)

cc @yeya24

[yeya24]

I am currently working on this issue already. As I mentioned here, some configs should be global, and some configs should be specific to different endpoints.

For multiple endpoints in the query frontend to work, my idea is to have two tripperware first.

1. Query range tripperware, the same as the current code. It has limits, step_align, split_interval, results_cache and retry.

2. Metadata tripperware, not sure about the naming. This tripperware handles label names, label values and series queriers and I think they can use the same middlewares. For caching, it would be better have a separate response cache config for this tripperware like --metadata.response-cache.

For flags, currently most of them have a query-range prefix, do we want to rename some of them to make them global? Otherwise it would be very confusing.

      --query-range.split-interval=24h  
                              Split queries by an interval and execute in parallel, 0 disables it.
      --query-range.max-retries-per-request=5  
                              Maximum number of retries for a single request; beyond this, the downstream error is returned.
      --query-range.max-query-length=0  
                              Limit the query time range (end - start time) in the query-frontend, 0 disables it.
      --query-range.max-query-parallelism=14  
                              Maximum number of queries will be scheduled in parallel by the frontend.
      --query-range.response-cache-max-freshness=1m  
                              Most recent allowed cacheable result, to prevent caching very recent results that might still be in flux.
      --query-range.response-cache-config-file=<file-path>  
                              Path to YAML file that contains response cache configuration.
      --query-range.response-cache-config=<content>  
                              Alternative to 'query-range.response-cache-config-file' flag (lower priority). Content of YAML file that contains response cache
                              configuration.

[yeya24]

Any ideas are welcomed and let's discuss these options in this pr :).