Change shuffle sharding ingester lookback default config

CHANGELOG.md

@@ -5,9 +5,10 @@
 ### Grafana Mimir
 
 * [CHANGE] Increased default configuration for `-server.grpc-max-recv-msg-size-bytes` and `-server.grpc-max-send-msg-size-bytes` from 4MB to 100MB. #1883
-* [CHANGE] Default values have changed for the following settings. This improves query performance for recent data (within 12h) by only reading from ingesters: #1909
+* [CHANGE] Default values have changed for the following settings. This improves query performance for recent data (within 12h) by only reading from ingesters: #1909 #1921
     - `-blocks-storage.bucket-store.ignore-blocks-within` now defaults to `10h` (previously `0`)
     - `-querier.query-store-after` now defaults to `12h` (previously `0`)
+    - `-querier.shuffle-sharding-ingesters-lookback-period` now defaults to `13h` (previously `0`)
 * [ENHANCEMENT] Store-gateway: Add the experimental ability to run requests in a dedicated OS thread pool. This feature can be configured using `-store-gateway.thread-pool-size` and is disabled by default. Replaces the ability to run index header operations in a dedicated thread pool. #1660 #1812
 * [ENHANCEMENT] Improved error messages to make them easier to understand and referencing a unique global identifier that can be looked up in the runbooks. #1907
 * [ENHANCEMENT] Memberlist KV: incoming messages are now processed on per-key goroutine. This may reduce loss of "maintanance" packets in busy memberlist installations, but use more CPU. New `memberlist_client_received_broadcasts_dropped_total` counter tracks number of dropped per-key messages. #1912
@@ -27,7 +28,7 @@
 
 ### Jsonnet
 
-* [CHANGE] Remove use of `-querier.query-store-after` and `-blocks-storage.bucket-store.ignore-blocks-within` CLI flags since the values now match defaults. #1915
+* [CHANGE] Remove use of `-querier.query-store-after`, `-querier.shuffle-sharding-ingesters-lookback-period`, `-blocks-storage.bucket-store.ignore-blocks-within`, and `-blocks-storage.tsdb.close-idle-tsdb-timeout` CLI flags since the values now match defaults. #1915 #1921
 
 ## 2.1.0-rc.0
 

cmd/mimir/config-descriptor.json

@@ -1480,9 +1480,9 @@
           "kind": "field",
           "name": "shuffle_sharding_ingesters_lookback_period",
           "required": false,
-          "desc": "When distributor's sharding strategy is shuffle-sharding and this setting is \u003e 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).",
+          "desc": "When this setting is \u003e 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).",
           "fieldValue": null,
-          "fieldDefaultValue": 0,
+          "fieldDefaultValue": 46800000000000,
           "fieldFlag": "querier.shuffle-sharding-ingesters-lookback-period",
           "fieldType": "duration",
           "fieldCategory": "advanced"

cmd/mimir/help-all.txt.tmpl

@@ -1086,7 +1086,7 @@ Usage of ./cmd/mimir/mimir:
   -querier.scheduler-address string
     	Address of the query-scheduler component, in host:port format. Only one of -querier.frontend-address or -querier.scheduler-address can be set. If neither is set, queries are only received via HTTP endpoint.
   -querier.shuffle-sharding-ingesters-lookback-period duration
-    	When distributor's sharding strategy is shuffle-sharding and this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled).
+    	When this setting is > 0, queriers fetch in-memory series from the minimum set of required ingesters, selecting only ingesters which may have received series since 'now - lookback period'. The lookback period should be greater or equal than the configured -querier.query-store-after and -querier.query-ingesters-within. If this setting is 0, queriers always query all ingesters (ingesters shuffle sharding on read path is disabled). (default 13h0m0s)
   -querier.store-gateway-client.tls-ca-path string
     	Path to the CA certificates file to validate server certificate against. If not set, the host's root CA certificates are used.
   -querier.store-gateway-client.tls-cert-path string

docs/sources/operators-guide/configuring/reference-configuration-parameters/index.md

@@ -835,15 +835,14 @@ store_gateway_client:
   # CLI flag: -querier.store-gateway-client.tls-insecure-skip-verify
   [tls_insecure_skip_verify: <boolean> | default = false]
 
-# (advanced) When distributor's sharding strategy is shuffle-sharding and this
-# setting is > 0, queriers fetch in-memory series from the minimum set of
-# required ingesters, selecting only ingesters which may have received series
-# since 'now - lookback period'. The lookback period should be greater or equal
-# than the configured -querier.query-store-after and
+# (advanced) When this setting is > 0, queriers fetch in-memory series from the
+# minimum set of required ingesters, selecting only ingesters which may have
+# received series since 'now - lookback period'. The lookback period should be
+# greater or equal than the configured -querier.query-store-after and
 # -querier.query-ingesters-within. If this setting is 0, queriers always query
 # all ingesters (ingesters shuffle sharding on read path is disabled).
 # CLI flag: -querier.shuffle-sharding-ingesters-lookback-period
-[shuffle_sharding_ingesters_lookback_period: <duration> | default = 0s]
+[shuffle_sharding_ingesters_lookback_period: <duration> | default = 13h]
 
 # The maximum number of concurrent queries. This config option should be set on
 # query-frontend too when query sharding is enabled.

operations/mimir-tests/test-defaults-generated.yaml

@@ -762,7 +762,6 @@ spec:
         - -querier.frontend-address=query-frontend-discovery.default.svc.cluster.local:9095
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
         - -server.grpc.keepalive.ping-without-stream-allowed=true
@@ -999,7 +998,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-disable-chunk-streaming-generated.yaml

@@ -850,7 +850,6 @@ spec:
         - -mem-ballast-size-bytes=268435456
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1073,7 +1072,6 @@ spec:
         - -ingester.ring.prefix=
         - -ingester.ring.replication-factor=3
         - -ingester.ring.store=consul
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1328,7 +1326,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-gossip-generated.yaml

@@ -883,7 +883,6 @@ spec:
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1110,7 +1109,6 @@ spec:
         - -memberlist.abort-if-join-fails=false
         - -memberlist.bind-port=7946
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1369,7 +1367,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-gossip-multi-zone-generated.yaml

@@ -1047,7 +1047,6 @@ spec:
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1321,7 +1320,6 @@ spec:
         - -memberlist.abort-if-join-fails=false
         - -memberlist.bind-port=7946
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1592,7 +1590,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true
@@ -1710,7 +1707,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true
@@ -1828,7 +1824,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-gossip-multikv-generated.yaml

@@ -895,7 +895,6 @@ spec:
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1128,7 +1127,6 @@ spec:
         - -memberlist.abort-if-join-fails=false
         - -memberlist.bind-port=7946
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1396,7 +1394,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-gossip-multikv-switch-primary-secondary-generated.yaml

@@ -895,7 +895,6 @@ spec:
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1128,7 +1127,6 @@ spec:
         - -memberlist.abort-if-join-fails=false
         - -memberlist.bind-port=7946
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1396,7 +1394,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-gossip-multikv-teardown-generated.yaml

@@ -886,7 +886,6 @@ spec:
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1113,7 +1112,6 @@ spec:
         - -memberlist.abort-if-join-fails=false
         - -memberlist.bind-port=7946
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1372,7 +1370,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-gossip-ruler-disabled-generated.yaml

@@ -865,7 +865,6 @@ spec:
         - -memberlist.join=gossip-ring.default.svc.cluster.local:7946
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1257,7 +1256,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true

operations/mimir-tests/test-multi-zone-generated.yaml

@@ -1001,7 +1001,6 @@ spec:
         - -mem-ballast-size-bytes=268435456
         - -querier.frontend-client.grpc-max-send-msg-size=104857600
         - -querier.max-concurrent=8
-        - -querier.query-ingesters-within=13h
         - -querier.scheduler-address=query-scheduler-discovery.default.svc.cluster.local:9095
         - -runtime-config.file=/etc/mimir/overrides.yaml
         - -server.grpc.keepalive.min-time-between-pings=10s
@@ -1271,7 +1270,6 @@ spec:
         - -ingester.ring.replication-factor=3
         - -ingester.ring.store=consul
         - -ingester.ring.zone-awareness-enabled=true
-        - -querier.query-ingesters-within=13h
         - -ruler-storage.backend=gcs
         - -ruler-storage.gcs.bucket-name=rules-bucket
         - -ruler.alertmanager-url=http://alertmanager.default.svc.cluster.local/alertmanager
@@ -1538,7 +1536,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true
@@ -1651,7 +1648,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true
@@ -1764,7 +1760,6 @@ spec:
         - -blocks-storage.backend=gcs
         - -blocks-storage.gcs.bucket-name=blocks-bucket
         - -blocks-storage.tsdb.block-ranges-period=2h
-        - -blocks-storage.tsdb.close-idle-tsdb-timeout=13h
         - -blocks-storage.tsdb.dir=/data/tsdb
         - -blocks-storage.tsdb.ship-interval=1m
         - -distributor.health-check-ingesters=true