batch and stream related metrics in store gateway

[kadhamecha-conga]

hi team,

is there metrics available for batch and stream related for store-Gateway ? or is there any other thing i can check related to memory usage in 2.7.1 mimir version?

thanks.

[dimitarvdimitrov]

the queries dashboard should have info about where time is spent in the store-gateway and which data type consumes the most memory

[kadhamecha-conga]

hi @dimitarvdimitrov, thanks for sharing information. it helps.

can you please also help me to understand below three things:

how can I improve batching efficiency ? ( this is currently set to streaming_series_batch_size =5000, as default )
how can I reduce stream latency?
resource planning for store gateway depends on which factor?

thanks.

[dimitarvdimitrov]

how can I improve batching efficiency ?

reducing the batch size will help here. This comes with the tradeoff of increased object store requests.

how can I reduce stream latency?

Increasing index and chunks caches replicas is one way to do it. The hit rate of those is in the Mimir / Reads dashboard. Try to aim for about 95-99% hit rate for the chunks cache and 80-90% hit rate for the index cache.

resource planning for store gateway depends on which factor?

It is mostly dependant on the rate of data that the store-gateway touches. From an internal cluster I can see about 1-1.5GiB of memory and 0.15-0.2 cores for each 1MiB of Data touched/sec. You can scale horizontally based on disk space (targeting e.g. 50% utilization) and scale vertically based on the data touched/sec.

edit: these resources are the total across 3 zones and replication factor 3.

[kadhamecha-conga]

thanks @dimitarvdimitrov, it helps.

the same thing also affect on querier, it is also killed by oom when I requesting for higher range of 30d.
i'm referring large.yaml where querier and store gateway are exact match on resource. can you please suggest on querier part also to tune it?

this is what touched and fetched data per sec graph.

as per graph, if data touched to 143 Mib/S, Does this mean store-gw combine cpu and memory requirement is 144GB and 28 core cpu ?

this helps to do capacity plan for read path.

can you please also confirm which version has below changes?
https://grafana.com/blog/2023/08/21/less-is-more-how-grafana-mimir-queries-run-faster-and-more-cost-efficiently-with-fewer-indexes/

thanks.

[dimitarvdimitrov]

If I'm reading the graph correctly it spikes to above 238MiB/s. This would mean 47 cores and 238-357 GiB of memory

[kadhamecha-conga]

@dimitarvdimitrov , I have RF=3 and sg running with below spec, but still three of SG pods killed by OOM.

replicas: 8
resources:
limits:
memory: 10Gi
requests:
cpu: 1
memory: 8Gi

data touched max is 28MiB and Sg does have those resources.

let me know, if i missing something

thanks.