Thanos 0.35.1 : receive ingestor + router configuration intermittently showing messages "Looked up series not found"

[chit786]

Setup information:

• thanos receiver ( routing + ingestor ) config with ketama hashring
• thanos receive controller is used

We are stress testing thanos and see following messages in the receiver logs which I tracked the source down to this , does receiver lookup for prometheus_tsdb_head_series ?

ts=2024-06-05T21:46:45.34461492Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"
ts=2024-06-05T21:46:45.344618477Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"
ts=2024-06-05T21:46:45.344622037Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"
ts=2024-06-05T21:46:45.344627549Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"
ts=2024-06-05T21:46:45.344631192Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"
ts=2024-06-05T21:46:45.344635337Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"
ts=2024-06-05T21:46:45.344638616Z caller=head_read.go:131 level=debug component=receive component=multi-tsdb tenant=default-tenant msg="Looked up series not found"

receiver ( ingestor ) config:

    - name: receive
      image: docker.io/bitnami/thanos:0.35.1-debian-12-r0
      args:
        - receive
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--grpc-address=0.0.0.0:10901'
        - '--http-address=0.0.0.0:10902'
        - '--remote-write.address=0.0.0.0:19291'
        - '--objstore.config=$(OBJSTORE_CONFIG)'
        - '--tsdb.path=/var/thanos/receive'
        - '--label=replica="$(NAME)"'
        - '--label=receive="true"'
        - '--tsdb.retention=6h'
        - >-
          --receive.local-endpoint=$(NAME).kp-shared-receive-headless.$(NAMESPACE).svc.cluster.local:10901
        - '--receive.hashrings-algorithm=ketama'
        - '--receive.hashrings-file-refresh-interval=5m'
        - '--receive.forward.async-workers=20000'
        - '--receive-forward-timeout=10m'
        - '--receive-forward-max-backoff=5s'
        - '--tsdb.memory-snapshot-on-shutdown'
        - '--tsdb.min-block-duration=1h'
        - '--tsdb.max-block-duration=1h'
        - '--writer.intern'

receive router config:

    - name: receive
      image: docker.io/bitnami/thanos:0.35.1-debian-12-r0
      args:
        - receive
        - '--log.level=debug'
        - '--log.format=logfmt'
        - '--grpc-address=0.0.0.0:10901'
        - '--http-address=0.0.0.0:10902'
        - '--remote-write.address=0.0.0.0:19291'
        - '--label=replica="$(NAME)"'
        - '--label=receive="true"'
        - '--receive.hashrings-file=/var/lib/thanos-receive/hashrings.json'
        - '--receive.replication-factor=3'
        - '--receive.hashrings-algorithm=ketama'
        - '--receive.hashrings-file-refresh-interval=5m'
        - '--receive.forward.async-workers=10000'
        - '--receive-forward-timeout=10m'
        - '--receive-forward-max-backoff=5s'