expanding series: failed to fetch some blocks - err-mimir-store-consistency-check-failed

[kadhamecha-conga]

hi team,

i'm getting this error in store-gateway as expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed).

I see no pvc size issue for store-gateway, lot of space is still left.

query-frontend logs are as:

ts=2023-10-13T14:16:19.977205199Z caller=retry.go:78 level=error user=anonymous msg="error processing request" try=0 err="rpc error: code = Code(500) desc = {\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: 01HCBMXFQ55KYYYP5J05BKS12H 01HCE4N16N6EVZJKZKKN74Z090\"}"

can you please help me to resolve it ?

mimir - 2.7.1

config for store-gateway:

store_gateway:
  persistentVolume:
    storageClass: gp2
    size: 50Gi
  replicas: 6
  resources:
    limits:
      memory: 10Gi
    requests:
      cpu: 1
      memory: 8Gi

thanks.

[kadhamecha-conga]

hi , i'm attaching meta.json file for below blocks.

01HCE4N16N6EVZJKZKKN74Z090- meta (4).json

01HCBMXFQ55KYYYP5J05BKS12H- meta (3).json

can someone please let me know, what's issue here ?

[Vaibhav-1995]

Hi , any update on this issue ?

internal: expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: XXXXXXXXXXXX

[dimitarvdimitrov]

failed consistency check usually points to unavailable store-gateways. I would suggest to check whether there aren't any alerts from the mixin which are firing and that all store-gateways are available and serving during this time. Querier and store-gateway logs during this time might be of help

[Vaibhav-1995]

I'm not usging mixin - configured s3 bucket and below are the logs from Querier and store-gateway

Querier -
ts=2024-04-08T05:19:05.57846861Z caller=logging.go:86 level=warn traceID=XXX orgID=anonymous msg="GET /prometheus/api/v1/query_range?end=1712534280&query=kube_pod_info+%2A+on+%28namespace%2C+pod%29+group_left+%28label_env%2C+label_svc%2C+label_app%29+kube_pod_labels%7Blabel_app%3D%22XXX%22%2Clabel_bu%3D%22corp%22%2Clabel_env%3D%22qa%22%2Clabel_svc%3D%22XXX%22%2Cpod%3D~%22XXXj%22%7D&start=1712510280&step=120 (500) 172.956338ms Response: "{\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: failed to fetch some blocks (err-mimir-store-consistency-check-failed). The failed blocks are: XXXXXXXXXX\"}" ws: false; Accept: application/json; X-Scope-Orgid: anonymous; uber-trace-id: XXX:XXX:XXX:0; "

Store-gateway -
ts=2024-04-08T05:08:37.247953219Z caller=bucket.go:410 level=warn user=anonymous msg="loading block failed" elapsed=543.073684ms id=XXXXXXXerr="create index header reader: write index header: 2 errors: copy symbols: write /data/tsdb-sync/anonymous/XXXXXXX/index-header.tmp: no space left on device; close binary writer for /data/tsdb-sync/anonymous/XXXXXXX/index-header.tmp: write /data/tsdb-sync/anonymous/XXX/index-header.tmp: no space left on device"
ts=2024-04-08T05:20:43.094731134Z caller=reporter.go:175 level=info msg="failed to send anonymous usage stats report" err="send the report to the stats server: Post "https://stats.grafana.org/mimir-usage-report\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

[dimitarvdimitrov]

no space left on device points to the store-gateway running out of disk space. I'd add more store-gateways

[Vaibhav-1995]

Thanks @dimitarvdimitrov
It worked perfectly