Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store panics with "slice bounds out of range" #3057

Closed
zltyfsh opened this issue Aug 19, 2020 · 7 comments
Closed

store panics with "slice bounds out of range" #3057

zltyfsh opened this issue Aug 19, 2020 · 7 comments
Labels
bug component: store difficulty: medium Hacktoberfest help wanted stale

Comments

@zltyfsh
Copy link

zltyfsh commented Aug 19, 2020

Thanos, Prometheus and Golang version used:

prometheus/prometheus:v2.19.0
thanos/thanos:v0.14.0

Object Storage Provider:

S3

What happened:

When running a query with a 90 days timespan, the store panics.

Full logs to relevant components:

Logs from store

level=debug ts=2020-08-19T07:50:51.943767339Z caller=main.go:102 msg="maxprocs: Leaving GOMAXPROCS=[2]: CPU quota undefined"
level=info ts=2020-08-19T07:50:51.943987303Z caller=main.go:151 msg="Tracing will be disabled"
level=info ts=2020-08-19T07:50:51.944096933Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2020-08-19T07:50:51.944459648Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
level=info ts=2020-08-19T07:50:51.944766235Z caller=options.go:23 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2020-08-19T07:50:51.944835531Z caller=grpc.go:95 service=gRPC/server component=store msg="registering as gRPC StoreAPI"
level=info ts=2020-08-19T07:50:51.945174296Z caller=store.go:374 msg="starting store node"
level=info ts=2020-08-19T07:50:51.945296817Z caller=store.go:321 msg="initializing bucket store"
level=info ts=2020-08-19T07:50:51.946285789Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2020-08-19T07:50:51.946304337Z caller=http.go:56 service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-08-19T07:50:52.504771373Z caller=fetcher.go:452 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=559.439362ms cached=65 returned=65 partial=0
level=debug ts=2020-08-19T07:50:52.506595828Z caller=bucket.go:447 msg="loading new block" id=01EFGC2D7YY1A38MY13AH8DHSA
level=debug ts=2020-08-19T07:50:52.506719259Z caller=bucket.go:447 msg="loading new block" id=01EF0Y4DX4XX57WEVFS39F8Z7R
level=debug ts=2020-08-19T07:50:52.507016245Z caller=bucket.go:447 msg="loading new block" id=01EFGBW4Z7P3P4WN2C7NYPXYMQ
level=debug ts=2020-08-19T07:50:52.507494206Z caller=bucket.go:447 msg="loading new block" id=01EBH42JX5JF0GC7PJD7FAWYQD
level=debug ts=2020-08-19T07:50:52.507939969Z caller=bucket.go:447 msg="loading new block" id=01ECRX14TAKB4PRF5RFAFAD158
level=debug ts=2020-08-19T07:50:52.508271659Z caller=bucket.go:447 msg="loading new block" id=01EFZT257VB1RMMQRPXNFTGGJE
level=debug ts=2020-08-19T07:50:52.508662329Z caller=bucket.go:447 msg="loading new block" id=01EF624YVVXF9C0NN5P5TYP9JF
level=debug ts=2020-08-19T07:50:52.509104397Z caller=bucket.go:447 msg="loading new block" id=01EG0ND1WNJK62PECHD3WGPVE7
level=debug ts=2020-08-19T07:50:52.524814346Z caller=bucket.go:447 msg="loading new block" id=01EFTNSY8VMBTGG9ND7ES64T00
level=debug ts=2020-08-19T07:50:52.589575134Z caller=bucket.go:447 msg="loading new block" id=01EG2R8WH1P0T4NF3DPZKD1XJ7
level=debug ts=2020-08-19T07:50:52.615187161Z caller=bucket.go:447 msg="loading new block" id=01EBH16G8D00ZG9YGVJ5M0WT00
level=debug ts=2020-08-19T07:50:52.702420313Z caller=bucket.go:447 msg="loading new block" id=01EBH2X92A39NQ4NXT3AFMCDSW
level=debug ts=2020-08-19T07:50:52.719935374Z caller=bucket.go:447 msg="loading new block" id=01EBH2BMRQ3E608Q4Y8XJT2YBX
level=debug ts=2020-08-19T07:50:52.811966185Z caller=bucket.go:447 msg="loading new block" id=01EG0NEBZNP1VWBQV9835PZJAK
level=debug ts=2020-08-19T07:50:52.852946541Z caller=bucket.go:447 msg="loading new block" id=01EF12ZH6J57QVEVBP6HHVMAF4
level=debug ts=2020-08-19T07:50:52.877977648Z caller=bucket.go:447 msg="loading new block" id=01EBH0TH8R18WW418YKMSGAN7Q
level=debug ts=2020-08-19T07:50:52.922345171Z caller=bucket.go:447 msg="loading new block" id=01EDX0018AXVXSFBFSS66YEJTG
level=debug ts=2020-08-19T07:50:52.958393211Z caller=bucket.go:447 msg="loading new block" id=01EBZ33FXX6GDCP2TARBGQZ2WG
level=debug ts=2020-08-19T07:50:52.993419192Z caller=bucket.go:447 msg="loading new block" id=01EDWZ6Y32JYQ4GD1V8YM8PZZM
level=debug ts=2020-08-19T07:50:53.011770279Z caller=bucket.go:447 msg="loading new block" id=01EF62H6CWMCB72ECX3D3WDT7R
level=info ts=2020-08-19T07:50:53.10390706Z caller=bucket.go:456 msg="loaded new block" elapsed=597.188299ms id=01EF0Y4DX4XX57WEVFS39F8Z7R
level=debug ts=2020-08-19T07:50:53.103946772Z caller=bucket.go:447 msg="loading new block" id=01EG1GVZFY060DNNSA8MB426J0
level=info ts=2020-08-19T07:50:53.1314428Z caller=bucket.go:456 msg="loaded new block" elapsed=623.50276ms id=01ECRX14TAKB4PRF5RFAFAD158
level=info ts=2020-08-19T07:50:53.146177727Z caller=bucket.go:456 msg="loaded new block" elapsed=639.581467ms id=01EFGC2D7YY1A38MY13AH8DHSA
level=debug ts=2020-08-19T07:50:53.186999376Z caller=bucket.go:447 msg="loading new block" id=01EG1GX9EAY08QGXB20567KV4T
level=info ts=2020-08-19T07:50:53.275573932Z caller=bucket.go:456 msg="loaded new block" elapsed=263.803306ms id=01EF62H6CWMCB72ECX3D3WDT7R
level=debug ts=2020-08-19T07:50:53.27561038Z caller=bucket.go:447 msg="loading new block" id=01EFNGSX20HD27586F9C0X4ACW
level=info ts=2020-08-19T07:50:53.189339161Z caller=bucket.go:456 msg="loaded new block" elapsed=377.37579ms id=01EG0NEBZNP1VWBQV9835PZJAK
level=info ts=2020-08-19T07:50:53.18952963Z caller=bucket.go:456 msg="loaded new block" elapsed=680.868165ms id=01EF624YVVXF9C0NN5P5TYP9JF
level=info ts=2020-08-19T07:50:53.199341016Z caller=bucket.go:456 msg="loaded new block" elapsed=674.527887ms id=01EFTNSY8VMBTGG9ND7ES64T00
level=info ts=2020-08-19T07:50:53.199451248Z caller=bucket.go:456 msg="loaded new block" elapsed=277.110318ms id=01EDX0018AXVXSFBFSS66YEJTG
level=info ts=2020-08-19T07:50:53.199565706Z caller=bucket.go:456 msg="loaded new block" elapsed=692.072481ms id=01EBH42JX5JF0GC7PJD7FAWYQD
level=info ts=2020-08-19T07:50:53.199784234Z caller=bucket.go:456 msg="loaded new block" elapsed=206.36834ms id=01EDWZ6Y32JYQ4GD1V8YM8PZZM
level=debug ts=2020-08-19T07:50:53.199796832Z caller=bucket.go:447 msg="loading new block" id=01EFNGF7Y6G5K0RQWMVP3KAHXS
level=debug ts=2020-08-19T07:50:53.288537643Z caller=bucket.go:447 msg="loading new block" id=01EFTNEX6TGNK2EXEDG1A9SAQZ
level=info ts=2020-08-19T07:50:53.199927175Z caller=bucket.go:456 msg="loaded new block" elapsed=610.372257ms id=01EG2R8WH1P0T4NF3DPZKD1XJ7
level=info ts=2020-08-19T07:50:53.20010179Z caller=bucket.go:456 msg="loaded new block" elapsed=241.712666ms id=01EBZ33FXX6GDCP2TARBGQZ2WG
level=debug ts=2020-08-19T07:50:53.374454716Z caller=bucket.go:447 msg="loading new block" id=01EFTNN8G749E9YT63QDCKZKFN
level=info ts=2020-08-19T07:50:53.37456053Z caller=bucket.go:456 msg="loaded new block" elapsed=496.585134ms id=01EBH0TH8R18WW418YKMSGAN7Q
level=info ts=2020-08-19T07:50:53.374952393Z caller=bucket.go:456 msg="loaded new block" elapsed=867.937256ms id=01EFGBW4Z7P3P4WN2C7NYPXYMQ
level=debug ts=2020-08-19T07:50:53.375048479Z caller=bucket.go:447 msg="loading new block" id=01ECRZ9WPCWG953SRD4WK88X5R
level=debug ts=2020-08-19T07:50:53.404629801Z caller=bucket.go:447 msg="loading new block" id=01EG2R8WH1QGD4QMB8WWWW57D7
level=debug ts=2020-08-19T07:50:53.412516139Z caller=bucket.go:447 msg="loading new block" id=01EBH4J65EPTPDS7M27ATTZKAE
level=debug ts=2020-08-19T07:50:53.424992836Z caller=bucket.go:447 msg="loading new block" id=01EFB7B06B8AZ17Y11GH47BWDK
level=debug ts=2020-08-19T07:50:53.449066939Z caller=bucket.go:447 msg="loading new block" id=01EFZTEDJMQ7AZZFBZDSBT3F94
level=info ts=2020-08-19T07:50:53.214434495Z caller=bucket.go:456 msg="loaded new block" elapsed=705.329492ms id=01EG0ND1WNJK62PECHD3WGPVE7
level=debug ts=2020-08-19T07:50:53.463522425Z caller=bucket.go:447 msg="loading new block" id=01EG2CC63C3X0E6AQBCX8XFVK8
level=debug ts=2020-08-19T07:50:53.491435459Z caller=bucket.go:447 msg="loading new block" id=01ECRZJAHRX6PR6TP9TH9FAN2K
level=debug ts=2020-08-19T07:50:53.494106615Z caller=bucket.go:447 msg="loading new block" id=01EG2AHE100N9T3BR3W30FM2BZ
level=info ts=2020-08-19T07:50:53.214587462Z caller=bucket.go:456 msg="loaded new block" elapsed=361.645485ms id=01EF12ZH6J57QVEVBP6HHVMAF4
level=debug ts=2020-08-19T07:50:53.517570331Z caller=bucket.go:447 msg="loading new block" id=01EG2HD590P5348M0GPWTJTJ54
level=debug ts=2020-08-19T07:50:53.531592954Z caller=bucket.go:447 msg="loading new block" id=01EBZ3886MNQ5HN31DE8W4FJ8S
level=info ts=2020-08-19T07:50:53.221017918Z caller=bucket.go:456 msg="loaded new block" elapsed=712.745296ms id=01EFZT257VB1RMMQRPXNFTGGJE
level=debug ts=2020-08-19T07:50:53.539512063Z caller=bucket.go:447 msg="loading new block" id=01ECRYAJWN6SRWD95MDH775AWX
level=info ts=2020-08-19T07:50:53.324666378Z caller=bucket.go:456 msg="loaded new block" elapsed=220.718907ms id=01EG1GVZFY060DNNSA8MB426J0
level=debug ts=2020-08-19T07:50:53.568440973Z caller=bucket.go:447 msg="loading new block" id=01EFTN8QSWJ1GYYJT02C86QG1Y
level=info ts=2020-08-19T07:50:53.374429589Z caller=bucket.go:456 msg="loaded new block" elapsed=654.49401ms id=01EBH2BMRQ3E608Q4Y8XJT2YBX
level=debug ts=2020-08-19T07:50:53.593883269Z caller=bucket.go:447 msg="loading new block" id=01EF10S0C08MJ7BP85B8TKVSRK
level=info ts=2020-08-19T07:50:53.62098699Z caller=bucket.go:456 msg="loaded new block" elapsed=433.984305ms id=01EG1GX9EAY08QGXB20567KV4T
level=debug ts=2020-08-19T07:50:53.623463131Z caller=bucket.go:447 msg="loading new block" id=01EF12QY6WC6RK68CYN66CSPHS
level=info ts=2020-08-19T07:50:53.641479259Z caller=bucket.go:456 msg="loaded new block" elapsed=441.681792ms id=01EFNGF7Y6G5K0RQWMVP3KAHXS
level=debug ts=2020-08-19T07:50:53.641641791Z caller=bucket.go:447 msg="loading new block" id=01EBZ3CWB3NQ0XETV76WK4C0VY
level=info ts=2020-08-19T07:50:53.674402543Z caller=bucket.go:456 msg="loaded new block" elapsed=385.865666ms id=01EFTNEX6TGNK2EXEDG1A9SAQZ
level=debug ts=2020-08-19T07:50:53.676709848Z caller=bucket.go:447 msg="loading new block" id=01EF11X87RB2E00RZKQF5RW4GK
level=info ts=2020-08-19T07:50:53.688020251Z caller=bucket.go:456 msg="loaded new block" elapsed=412.409845ms id=01EFNGSX20HD27586F9C0X4ACW
level=info ts=2020-08-19T07:50:53.693186276Z caller=bucket.go:456 msg="loaded new block" elapsed=175.617208ms id=01EG2HD590P5348M0GPWTJTJ54
level=info ts=2020-08-19T07:50:53.69330803Z caller=bucket.go:456 msg="loaded new block" elapsed=268.319201ms id=01EFB7B06B8AZ17Y11GH47BWDK
level=info ts=2020-08-19T07:50:53.69342093Z caller=bucket.go:456 msg="loaded new block" elapsed=199.3169ms id=01EG2AHE100N9T3BR3W30FM2BZ
level=debug ts=2020-08-19T07:50:53.694773445Z caller=bucket.go:447 msg="loading new block" id=01EFB6YJC4TKQXPZ3HW4VCW6JZ
level=info ts=2020-08-19T07:50:53.699634286Z caller=bucket.go:456 msg="loaded new block" elapsed=287.120711ms id=01EBH4J65EPTPDS7M27ATTZKAE
level=debug ts=2020-08-19T07:50:53.699660859Z caller=bucket.go:447 msg="loading new block" id=01EG2AHE10DTYFVEH3DHH3M8AZ
level=debug ts=2020-08-19T07:50:53.70593222Z caller=bucket.go:447 msg="loading new block" id=01EFB74NJV4XPCV3GHS055ZR8H
level=debug ts=2020-08-19T07:50:53.717390317Z caller=bucket.go:447 msg="loading new block" id=01EF0ZBGHXNXF1TA4HAAXKAWTS
level=info ts=2020-08-19T07:50:53.720960094Z caller=bucket.go:456 msg="loaded new block" elapsed=271.897988ms id=01EFZTEDJMQ7AZZFBZDSBT3F94
level=debug ts=2020-08-19T07:50:53.72098705Z caller=bucket.go:447 msg="loading new block" id=01EF62B02AP7ZMBKK2B51RP4ZC
level=info ts=2020-08-19T07:50:53.744017167Z caller=bucket.go:456 msg="loaded new block" elapsed=280.496949ms id=01EG2CC63C3X0E6AQBCX8XFVK8
level=debug ts=2020-08-19T07:50:53.744047798Z caller=bucket.go:447 msg="loading new block" id=01EFNGYET47NQJ9XC0JJKGK1MT
level=debug ts=2020-08-19T07:50:53.744626527Z caller=bucket.go:447 msg="loading new block" id=01EBGGYSXNXC056QDPKRC70SHY
level=info ts=2020-08-19T07:50:53.835978817Z caller=bucket.go:456 msg="loaded new block" elapsed=461.523267ms id=01EFTNN8G749E9YT63QDCKZKFN
level=debug ts=2020-08-19T07:50:53.83601321Z caller=bucket.go:447 msg="loading new block" id=01EFZTK3DBATG45H2HM4AXQY4W
level=info ts=2020-08-19T07:50:53.840584891Z caller=bucket.go:456 msg="loaded new block" elapsed=301.074204ms id=01ECRYAJWN6SRWD95MDH775AWX
level=info ts=2020-08-19T07:50:53.84087435Z caller=bucket.go:456 msg="loaded new block" elapsed=199.233608ms id=01EBZ3CWB3NQ0XETV76WK4C0VY
level=info ts=2020-08-19T07:50:53.856740476Z caller=bucket.go:456 msg="loaded new block" elapsed=325.147666ms id=01EBZ3886MNQ5HN31DE8W4FJ8S
level=debug ts=2020-08-19T07:50:53.856995802Z caller=bucket.go:447 msg="loading new block" id=01EBH3JC54M8N56ZKF2X3D21Q9
level=info ts=2020-08-19T07:50:53.885781322Z caller=bucket.go:456 msg="loaded new block" elapsed=481.154706ms id=01EG2R8WH1QGD4QMB8WWWW57D7
level=debug ts=2020-08-19T07:50:53.88595179Z caller=bucket.go:447 msg="loading new block" id=01EG2HD591W8ZTB1ABFJ4YNHW9
level=info ts=2020-08-19T07:50:53.906607179Z caller=bucket.go:456 msg="loaded new block" elapsed=531.558881ms id=01ECRZ9WPCWG953SRD4WK88X5R
level=debug ts=2020-08-19T07:50:53.906699091Z caller=bucket.go:447 msg="loading new block" id=01EFNGME0DK44RFC5XDQQJ7G7P
level=debug ts=2020-08-19T07:50:53.921843715Z caller=bucket.go:447 msg="loading new block" id=01EFZT7VE11QAY3SCQ55G3PE2Q
level=debug ts=2020-08-19T07:50:53.932355659Z caller=bucket.go:447 msg="loading new block" id=01EDX06YDNTQZ2FZRV0251EX0V
level=info ts=2020-08-19T07:50:53.953768101Z caller=bucket.go:456 msg="loaded new block" elapsed=462.33796ms id=01ECRZJAHRX6PR6TP9TH9FAN2K
level=debug ts=2020-08-19T07:50:53.953844445Z caller=bucket.go:447 msg="loading new block" id=01EFB7GESKHMJCAAQD0JS0RNPZ
level=info ts=2020-08-19T07:50:53.96395423Z caller=bucket.go:456 msg="loaded new block" elapsed=219.905771ms id=01EFNGYET47NQJ9XC0JJKGK1MT
level=info ts=2020-08-19T07:50:53.964513739Z caller=bucket.go:456 msg="loaded new block" elapsed=243.526835ms id=01EF62B02AP7ZMBKK2B51RP4ZC
level=debug ts=2020-08-19T07:50:53.968227911Z caller=bucket.go:447 msg="loading new block" id=01EFGC7GS54745AV2RR203BBWG
level=info ts=2020-08-19T07:50:53.972601647Z caller=bucket.go:456 msg="loaded new block" elapsed=349.140246ms id=01EF12QY6WC6RK68CYN66CSPHS
level=debug ts=2020-08-19T07:50:53.972686013Z caller=bucket.go:447 msg="loading new block" id=01EG2CAVE27G1PSE0277K2G6MC
level=info ts=2020-08-19T07:50:54.008521119Z caller=bucket.go:456 msg="loaded new block" elapsed=308.861167ms id=01EG2AHE10DTYFVEH3DHH3M8AZ
level=info ts=2020-08-19T07:50:54.008982931Z caller=bucket.go:456 msg="loaded new block" elapsed=314.210816ms id=01EFB6YJC4TKQXPZ3HW4VCW6JZ
level=debug ts=2020-08-19T07:50:54.009174732Z caller=bucket.go:447 msg="loading new block" id=01EDWXANKH0P92CZHZNM29C4BT
level=info ts=2020-08-19T07:50:54.011020751Z caller=bucket.go:456 msg="loaded new block" elapsed=334.311587ms id=01EF11X87RB2E00RZKQF5RW4GK
level=debug ts=2020-08-19T07:50:54.011055736Z caller=bucket.go:447 msg="loading new block" id=01EBH1VNPY6ZVMRR1YAPTCA4NZ
level=debug ts=2020-08-19T07:50:54.024297003Z caller=bucket.go:447 msg="loading new block" id=01EDWWEYKD8J8JEB74FGHER9F8
level=info ts=2020-08-19T07:50:54.047315095Z caller=bucket.go:456 msg="loaded new block" elapsed=341.383719ms id=01EFB74NJV4XPCV3GHS055ZR8H
level=debug ts=2020-08-19T07:50:54.047390956Z caller=bucket.go:447 msg="loading new block" id=01EF62Q208MDY02NJ5WTH4492F
level=info ts=2020-08-19T07:50:54.100634876Z caller=bucket.go:456 msg="loaded new block" elapsed=356.008441ms id=01EBGGYSXNXC056QDPKRC70SHY
level=debug ts=2020-08-19T07:50:54.100676332Z caller=bucket.go:447 msg="loading new block" id=01EDWY55E3BPE1TAS4BZ1HDD23
level=info ts=2020-08-19T07:50:54.101636783Z caller=bucket.go:456 msg="loaded new block" elapsed=384.247048ms id=01EF0ZBGHXNXF1TA4HAAXKAWTS
level=debug ts=2020-08-19T07:50:54.101683817Z caller=bucket.go:447 msg="loading new block" id=01EFGBNP866SD09K2SKDSNP814
level=debug ts=2020-08-19T07:50:54.13555032Z caller=bucket.go:447 msg="loading new block" id=01EBZ3KNA5AAN7SQ7EQ8B0WTFT
level=info ts=2020-08-19T07:50:54.141045803Z caller=bucket.go:456 msg="loaded new block" elapsed=1.525863355s id=01EBH16G8D00ZG9YGVJ5M0WT00
level=info ts=2020-08-19T07:50:54.146418634Z caller=bucket.go:456 msg="loaded new block" elapsed=310.405398ms id=01EFZTK3DBATG45H2HM4AXQY4W
level=info ts=2020-08-19T07:50:54.150501555Z caller=bucket.go:456 msg="loaded new block" elapsed=1.448085781s id=01EBH2X92A39NQ4NXT3AFMCDSW
level=info ts=2020-08-19T07:50:54.150708679Z caller=bucket.go:456 msg="loaded new block" elapsed=582.27168ms id=01EFTN8QSWJ1GYYJT02C86QG1Y
level=info ts=2020-08-19T07:50:54.150824968Z caller=bucket.go:456 msg="loaded new block" elapsed=264.87509ms id=01EG2HD591W8ZTB1ABFJ4YNHW9
level=info ts=2020-08-19T07:50:54.151034311Z caller=bucket.go:456 msg="loaded new block" elapsed=557.153849ms id=01EF10S0C08MJ7BP85B8TKVSRK
level=info ts=2020-08-19T07:50:54.151223652Z caller=bucket.go:456 msg="loaded new block" elapsed=229.381819ms id=01EFZT7VE11QAY3SCQ55G3PE2Q
level=info ts=2020-08-19T07:50:54.151449461Z caller=bucket.go:456 msg="loaded new block" elapsed=244.751765ms id=01EFNGME0DK44RFC5XDQQJ7G7P
level=info ts=2020-08-19T07:50:54.162570225Z caller=bucket.go:456 msg="loaded new block" elapsed=230.215504ms id=01EDX06YDNTQZ2FZRV0251EX0V
level=info ts=2020-08-19T07:50:54.185051976Z caller=bucket.go:456 msg="loaded new block" elapsed=160.756161ms id=01EDWWEYKD8J8JEB74FGHER9F8
level=info ts=2020-08-19T07:50:54.187497506Z caller=bucket.go:456 msg="loaded new block" elapsed=178.325134ms id=01EDWXANKH0P92CZHZNM29C4BT
level=info ts=2020-08-19T07:50:54.187702272Z caller=bucket.go:456 msg="loaded new block" elapsed=176.647197ms id=01EBH1VNPY6ZVMRR1YAPTCA4NZ
level=info ts=2020-08-19T07:50:54.187891773Z caller=bucket.go:456 msg="loaded new block" elapsed=330.897783ms id=01EBH3JC54M8N56ZKF2X3D21Q9
level=info ts=2020-08-19T07:50:54.188042717Z caller=bucket.go:456 msg="loaded new block" elapsed=215.357882ms id=01EG2CAVE27G1PSE0277K2G6MC
level=info ts=2020-08-19T07:50:54.188280023Z caller=bucket.go:456 msg="loaded new block" elapsed=234.436142ms id=01EFB7GESKHMJCAAQD0JS0RNPZ
level=info ts=2020-08-19T07:50:54.188510622Z caller=bucket.go:456 msg="loaded new block" elapsed=141.120551ms id=01EF62Q208MDY02NJ5WTH4492F
level=info ts=2020-08-19T07:50:54.194695279Z caller=bucket.go:456 msg="loaded new block" elapsed=59.148912ms id=01EBZ3KNA5AAN7SQ7EQ8B0WTFT
level=info ts=2020-08-19T07:50:54.200014505Z caller=bucket.go:456 msg="loaded new block" elapsed=99.338567ms id=01EDWY55E3BPE1TAS4BZ1HDD23
level=info ts=2020-08-19T07:50:54.200238131Z caller=bucket.go:456 msg="loaded new block" elapsed=232.012009ms id=01EFGC7GS54745AV2RR203BBWG
level=info ts=2020-08-19T07:50:54.202487145Z caller=bucket.go:456 msg="loaded new block" elapsed=100.804122ms id=01EFGBNP866SD09K2SKDSNP814
level=info ts=2020-08-19T07:50:54.203133051Z caller=store.go:327 msg="bucket store ready" init_duration=2.257809831s
level=info ts=2020-08-19T07:50:54.203574872Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2020-08-19T07:50:54.203805911Z caller=grpc.go:119 service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
level=info ts=2020-08-19T07:50:54.584757972Z caller=fetcher.go:452 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=381.537624ms cached=65 returned=65 partial=0
level=debug ts=2020-08-19T07:51:52.456123874Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=14 MaximumResolution=0 mint=1590047513000 maxt=1597823513000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1594252800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:52.456200624Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=14 MaximumResolution=0 mint=1590047513000 maxt=1597823513000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1594252800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:52.456123748Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=14 MaximumResolution=0 mint=1590047513000 maxt=1597823513000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1594252800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:52.456310966Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=14 MaximumResolution=0 mint=1590047513000 maxt=1597823513000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1594252800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:56.416105698Z caller=bucket.go:953 msg="stats query processed" stats="&{blocksQueried:28 postingsTouched:56 postingsTouchedSizeSum:212940 postingsToFetch:0 postingsFetched:56 postingsFetchedSizeSum:212940 postingsFetchCount:56 postingsFetchDurationSum:98990213662 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:212940 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:260 seriesTouchedSizeSum:175437 seriesFetched:260 seriesFetchedSizeSum:1990000 seriesFetchCount:28 seriesFetchDurationSum:22722666638 chunksTouched:27694 chunksTouchedSizeSum:1331104 chunksFetched:27694 chunksFetchedSizeSum:1943760 chunksFetchCount:28 chunksFetchDurationSum:14787406702 getAllDuration:3918045556 mergedSeriesCount:78 mergedChunksCount:27694 mergeDuration:41601119}" err=null
level=debug ts=2020-08-19T07:51:56.426899148Z caller=bucket.go:953 msg="stats query processed" stats="&{blocksQueried:28 postingsTouched:84 postingsTouchedSizeSum:213340 postingsToFetch:0 postingsFetched:84 postingsFetchedSizeSum:213340 postingsFetchCount:84 postingsFetchDurationSum:145984415721 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:213340 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:72 seriesTouchedSizeSum:50041 seriesFetched:68 seriesFetchedSizeSum:1730448 seriesFetchCount:26 seriesFetchDurationSum:21150546096 chunksTouched:7912 chunksTouchedSizeSum:380266 chunksFetched:7912 chunksFetchedSizeSum:874288 chunksFetchCount:28 chunksFetchDurationSum:16534067267 getAllDuration:3962933706 mergedSeriesCount:20 mergedChunksCount:7912 mergeDuration:7624394}" err=null
level=debug ts=2020-08-19T07:51:56.838254907Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=14 MaximumResolution=0 mint=1590047517000 maxt=1597823517000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1594252800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:56.838613404Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=14 MaximumResolution=0 mint=1590047517000 maxt=1597823517000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1594252800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.234430184Z caller=bucket.go:953 msg="stats query processed" stats="&{blocksQueried:28 postingsTouched:84 postingsTouchedSizeSum:213340 postingsToFetch:0 postingsFetched:0 postingsFetchedSizeSum:0 postingsFetchCount:0 postingsFetchDurationSum:0 cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 cachedPostingsOriginalSizeSum:0 cachedPostingsCompressedSizeSum:0 cachedPostingsCompressionTimeSum:0 cachedPostingsDecompressions:0 cachedPostingsDecompressionErrors:0 cachedPostingsDecompressionTimeSum:0 seriesTouched:72 seriesTouchedSizeSum:50041 seriesFetched:0 seriesFetchedSizeSum:0 seriesFetchCount:0 seriesFetchDurationSum:0 chunksTouched:7912 chunksTouchedSizeSum:380266 chunksFetched:7912 chunksFetchedSizeSum:874288 chunksFetchCount:28 chunksFetchDurationSum:8713060253 getAllDuration:390702925 mergedSeriesCount:20 mergedChunksCount:7912 mergeDuration:4812227}" err=null
level=debug ts=2020-08-19T07:51:57.408821434Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.409006823Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.40913091Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.409360742Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.421500579Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590040800000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.42169635Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590040800000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.422725241Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.423234182Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.452590173Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590040800000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.45266288Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590040800000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.459673472Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590040800000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.459737836Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590040800000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.460413707Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.46052739Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.493028916Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
level=debug ts=2020-08-19T07:51:57.493120666Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"

panic: runtime error: slice bounds out of range [41456:32288]

goroutine 14848 [running]:
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).loadSeries(0xc0026c9bc0, 0x266a460, 0xc002ab0000, 0xc004618b00, 0x15, 0x20, 0x100010000, 0x9f81e0, 0xa91b80, 0xc0055f3740, ...)
        /go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1851 +0x9d5
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).PreloadSeries.func2(0x1, 0x0)
        /go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1829 +0x9d
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0065d0c00, 0xc0055f3800)
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:54 +0x66

Blocks in S3 bucket

|            ULID            |        FROM         |        UNTIL        |  RANGE   | UNTIL-DOWN |  #SERIES  |    #SAMPLES    |   #CHUNKS   | COMP-LEVEL | COMP-FAILED |                 LABELS                 | RESOLUTION |  SOURCE   |
|----------------------------|---------------------|---------------------|----------|------------|-----------|----------------|-------------|------------|-------------|----------------------------------------|------------|-----------|
| 01EBH2BMRQ3E608Q4Y8XJT2YBX | 05-03-2020 06:00:00 | 25-03-2020 12:00:00 | 486h0m0s | -          | 866,859   | 159,364,273    | 1,889,609   | 6          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EBH3JC54M8N56ZKF2X3D21Q9 | 05-03-2020 06:00:00 | 25-03-2020 12:00:00 | 486h0m0s | -          | 874,578   | 159,241,453    | 1,897,195   | 6          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EBH1VNPY6ZVMRR1YAPTCA4NZ | 25-03-2020 12:00:00 | 14-04-2020 18:00:00 | 486h0m0s | -          | 1,145,001 | 148,393,816    | 2,116,348   | 6          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EBH42JX5JF0GC7PJD7FAWYQD | 25-03-2020 12:00:00 | 14-04-2020 18:00:00 | 486h0m0s | -          | 1,145,982 | 148,394,322    | 2,117,329   | 6          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EBH16G8D00ZG9YGVJ5M0WT00 | 14-04-2020 18:00:00 | 05-05-2020 00:00:00 | 486h0m0s | -          | 3,863,083 | 166,213,892    | 4,866,300   | 6          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EBH2X92A39NQ4NXT3AFMCDSW | 14-04-2020 18:00:00 | 05-05-2020 00:00:00 | 486h0m0s | -          | 3,837,555 | 166,190,142    | 4,840,778   | 6          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EBH4J65EPTPDS7M27ATTZKAE | 05-05-2020 00:00:00 | 25-05-2020 06:00:00 | 486h0m0s | -          | 1,023,256 | 129,231,813    | 1,708,425   | 6          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EBGGYSXNXC056QDPKRC70SHY | 25-05-2020 06:00:00 | 14-06-2020 12:00:00 | 486h0m0s | -          | 503,800   | 82,340,418     | 1,162,840   | 6          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EBH0TH8R18WW418YKMSGAN7Q | 25-05-2020 06:00:00 | 14-06-2020 12:00:00 | 486h0m0s | -          | 501,147   | 82,351,990     | 1,160,187   | 6          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EBZ33FXX6GDCP2TARBGQZ2WG | 14-06-2020 12:00:00 | 25-06-2020 00:00:00 | 252h0m0s | -12h0m0s   | 531,264   | 524,733,130    | 4,616,813   | 5          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EBZ3886MNQ5HN31DE8W4FJ8S | 14-06-2020 12:00:00 | 25-06-2020 00:00:00 | 252h0m0s | -12h0m0s   | 554,317   | 525,077,140    | 4,639,857   | 5          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EBZ3CWB3NQ0XETV76WK4C0VY | 14-06-2020 12:00:00 | 25-06-2020 00:00:00 | 252h0m0s | -          | 554,317   | 44,446,240     | 843,192     | 5          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EBZ3KNA5AAN7SQ7EQ8B0WTFT | 14-06-2020 12:00:00 | 25-06-2020 00:00:00 | 252h0m0s | -          | 531,264   | 44,422,944     | 820,137     | 5          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01ECRX14TAKB4PRF5RFAFAD158 | 25-06-2020 00:00:00 | 09-07-2020 00:00:00 | 336h0m0s | -96h0m0s   | 385,829   | 688,264,621    | 5,138,483   | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01ECRYAJWN6SRWD95MDH775AWX | 25-06-2020 00:00:00 | 09-07-2020 00:00:00 | 336h0m0s | -96h0m0s   | 383,466   | 688,206,288    | 5,136,101   | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01ECRZ9WPCWG953SRD4WK88X5R | 25-06-2020 00:00:00 | 09-07-2020 00:00:00 | 336h0m0s | -          | 385,829   | 58,231,517     | 897,181     | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01ECRZJAHRX6PR6TP9TH9FAN2K | 25-06-2020 00:00:00 | 09-07-2020 00:00:00 | 336h0m0s | -          | 383,466   | 58,091,602     | 894,818     | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EDWWEYKD8J8JEB74FGHER9F8 | 09-07-2020 00:00:00 | 23-07-2020 00:00:00 | 336h0m0s | -296h0m0s  | 246,910   | 13,743,956,756 | 114,509,493 | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EDWXANKH0P92CZHZNM29C4BT | 09-07-2020 00:00:00 | 23-07-2020 00:00:00 | 336h0m0s | -296h0m0s  | 263,672   | 13,798,494,350 | 115,029,079 | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EDWY55E3BPE1TAS4BZ1HDD23 | 09-07-2020 00:00:00 | 23-07-2020 00:00:00 | 336h0m0s | -96h0m0s   | 246,910   | 688,415,022    | 5,058,812   | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EDWZ6Y32JYQ4GD1V8YM8PZZM | 09-07-2020 00:00:00 | 23-07-2020 00:00:00 | 336h0m0s | -96h0m0s   | 263,672   | 690,682,725    | 5,075,579   | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EDX0018AXVXSFBFSS66YEJTG | 09-07-2020 00:00:00 | 23-07-2020 00:00:00 | 336h0m0s | -          | 263,672   | 58,145,063     | 749,916     | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EDX06YDNTQZ2FZRV0251EX0V | 09-07-2020 00:00:00 | 23-07-2020 00:00:00 | 336h0m0s | -          | 246,910   | 58,127,040     | 733,154     | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EF0Y4DX4XX57WEVFS39F8Z7R | 23-07-2020 00:00:00 | 06-08-2020 00:00:00 | 336h0m0s | -296h0m0s  | 228,653   | 13,998,062,813 | 116,661,012 | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EF0ZBGHXNXF1TA4HAAXKAWTS | 23-07-2020 00:00:00 | 06-08-2020 00:00:00 | 336h0m0s | -296h0m0s  | 229,520   | 13,998,824,066 | 116,669,419 | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EF10S0C08MJ7BP85B8TKVSRK | 23-07-2020 00:00:00 | 06-08-2020 00:00:00 | 336h0m0s | -96h0m0s   | 229,520   | 700,235,986    | 5,087,991   | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EF11X87RB2E00RZKQF5RW4GK | 23-07-2020 00:00:00 | 06-08-2020 00:00:00 | 336h0m0s | -96h0m0s   | 228,653   | 700,234,397    | 5,087,156   | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EF12QY6WC6RK68CYN66CSPHS | 23-07-2020 00:00:00 | 06-08-2020 00:00:00 | 336h0m0s | -          | 229,520   | 58,438,276     | 749,936     | 4          | false       | atlas_cluster=stm,replica=prometheus-0 | 1h0m0s     | compactor |
| 01EF12ZH6J57QVEVBP6HHVMAF4 | 23-07-2020 00:00:00 | 06-08-2020 00:00:00 | 336h0m0s | -          | 228,653   | 58,435,603     | 749,069     | 4          | false       | atlas_cluster=stm,replica=prometheus-1 | 1h0m0s     | compactor |
| 01EF624YVVXF9C0NN5P5TYP9JF | 06-08-2020 00:00:00 | 08-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 331,707   | 1,917,359,814  | 16,025,672  | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EF62B02AP7ZMBKK2B51RP4ZC | 06-08-2020 00:00:00 | 08-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 330,735   | 1,914,618,656  | 16,064,641  | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EF62H6CWMCB72ECX3D3WDT7R | 06-08-2020 00:00:00 | 08-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 331,707   | 96,121,390     | 924,886     | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EF62Q208MDY02NJ5WTH4492F | 06-08-2020 00:00:00 | 08-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 330,735   | 96,117,149     | 923,915     | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EFB6YJC4TKQXPZ3HW4VCW6JZ | 08-08-2020 00:00:00 | 10-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 170,965   | 1,901,287,609  | 15,850,246  | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EFB74NJV4XPCV3GHS055ZR8H | 08-08-2020 00:00:00 | 10-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 171,132   | 1,901,287,668  | 15,850,432  | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EFB7B06B8AZ17Y11GH47BWDK | 08-08-2020 00:00:00 | 10-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 171,132   | 95,070,915     | 831,023     | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EFB7GESKHMJCAAQD0JS0RNPZ | 08-08-2020 00:00:00 | 10-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 170,965   | 95,070,769     | 830,856     | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EFGBNP866SD09K2SKDSNP814 | 10-08-2020 00:00:00 | 12-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 437,784   | 1,893,692,666  | 15,917,528  | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EFGBW4Z7P3P4WN2C7NYPXYMQ | 10-08-2020 00:00:00 | 12-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 441,502   | 1,894,909,490  | 15,827,069  | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EFGC2D7YY1A38MY13AH8DHSA | 10-08-2020 00:00:00 | 12-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 441,502   | 95,162,482     | 996,857     | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EFGC7GS54745AV2RR203BBWG | 10-08-2020 00:00:00 | 12-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 437,784   | 95,154,087     | 993,150     | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EFNGF7Y6G5K0RQWMVP3KAHXS | 12-08-2020 00:00:00 | 14-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 189,865   | 1,909,324,968  | 15,927,458  | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EFNGME0DK44RFC5XDQQJ7G7P | 12-08-2020 00:00:00 | 14-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 190,205   | 1,909,327,308  | 15,927,854  | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EFNGSX20HD27586F9C0X4ACW | 12-08-2020 00:00:00 | 14-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 189,865   | 95,491,365     | 848,900     | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EFNGYET47NQJ9XC0JJKGK1MT | 12-08-2020 00:00:00 | 14-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 190,205   | 95,491,677     | 849,240     | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EFTN8QSWJ1GYYJT02C86QG1Y | 14-08-2020 00:00:00 | 16-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 179,767   | 1,957,825,157  | 16,324,443  | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EFTNEX6TGNK2EXEDG1A9SAQZ | 14-08-2020 00:00:00 | 16-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 179,925   | 1,957,824,979  | 16,324,633  | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EFTNN8G749E9YT63QDCKZKFN | 14-08-2020 00:00:00 | 16-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 179,767   | 97,902,391     | 858,896     | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EFTNSY8VMBTGG9ND7ES64T00 | 14-08-2020 00:00:00 | 16-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 179,925   | 97,902,701     | 859,054     | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EFZT257VB1RMMQRPXNFTGGJE | 16-08-2020 00:00:00 | 18-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 187,555   | 1,968,026,526  | 16,412,287  | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EFZT7VE11QAY3SCQ55G3PE2Q | 16-08-2020 00:00:00 | 18-08-2020 00:00:00 | 48h0m0s  | -8h0m0s    | 187,717   | 1,968,027,871  | 16,412,515  | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EFZTEDJMQ7AZZFBZDSBT3F94 | 16-08-2020 00:00:00 | 18-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 187,717   | 98,417,817     | 869,316     | 3          | false       | atlas_cluster=stm,replica=prometheus-0 | 5m0s       | compactor |
| 01EFZTK3DBATG45H2HM4AXQY4W | 16-08-2020 00:00:00 | 18-08-2020 00:00:00 | 48h0m0s  | 192h0m0s   | 187,555   | 98,417,414     | 869,139     | 3          | false       | atlas_cluster=stm,replica=prometheus-1 | 5m0s       | compactor |
| 01EG0ND1WNJK62PECHD3WGPVE7 | 18-08-2020 00:00:00 | 18-08-2020 08:00:00 | 8h0m0s   | 32h0m0s    | 177,205   | 332,462,377    | 2,773,825   | 2          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EG0NEBZNP1VWBQV9835PZJAK | 18-08-2020 00:00:00 | 18-08-2020 08:00:00 | 8h0m0s   | 32h0m0s    | 177,227   | 332,462,374    | 2,773,880   | 2          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EG1GVZFY060DNNSA8MB426J0 | 18-08-2020 08:00:00 | 18-08-2020 16:00:00 | 8h0m0s   | 32h0m0s    | 178,134   | 332,694,371    | 2,781,779   | 2          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EG1GX9EAY08QGXB20567KV4T | 18-08-2020 08:00:00 | 18-08-2020 16:00:00 | 8h0m0s   | 32h0m0s    | 178,120   | 332,795,619    | 2,781,765   | 2          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EG2CAVE27G1PSE0277K2G6MC | 18-08-2020 16:00:00 | 19-08-2020 00:00:00 | 8h0m0s   | 32h0m0s    | 174,845   | 333,819,189    | 2,782,703   | 2          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | compactor |
| 01EG2CC63C3X0E6AQBCX8XFVK8 | 18-08-2020 16:00:00 | 19-08-2020 00:00:00 | 8h0m0s   | 32h0m0s    | 174,842   | 333,819,153    | 2,782,699   | 2          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | compactor |
| 01EG2AHE100N9T3BR3W30FM2BZ | 19-08-2020 00:00:00 | 19-08-2020 02:00:00 | 2h0m0s   | 38h0m0s    | 175,645   | 83,613,061     | 697,810     | 1          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | sidecar   |
| 01EG2AHE10DTYFVEH3DHH3M8AZ | 19-08-2020 00:00:00 | 19-08-2020 02:00:00 | 2h0m0s   | 38h0m0s    | 175,644   | 83,613,049     | 697,841     | 1          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | sidecar   |
| 01EG2HD590P5348M0GPWTJTJ54 | 19-08-2020 02:00:00 | 19-08-2020 04:00:00 | 2h0m0s   | 38h0m0s    | 174,364   | 83,459,839     | 695,913     | 1          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | sidecar   |
| 01EG2HD591W8ZTB1ABFJ4YNHW9 | 19-08-2020 02:00:00 | 19-08-2020 04:00:00 | 2h0m0s   | 38h0m0s    | 174,363   | 83,459,775     | 695,912     | 1          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | sidecar   |
| 01EG2R8WH1P0T4NF3DPZKD1XJ7 | 19-08-2020 04:00:00 | 19-08-2020 06:00:00 | 2h0m0s   | 38h0m0s    | 173,919   | 83,448,424     | 695,469     | 1          | false       | atlas_cluster=stm,replica=prometheus-0 | 0s         | sidecar   |
| 01EG2R8WH1QGD4QMB8WWWW57D7 | 19-08-2020 04:00:00 | 19-08-2020 06:00:00 | 2h0m0s   | 38h0m0s    | 173,920   | 83,448,443     | 695,470     | 1          | false       | atlas_cluster=stm,replica=prometheus-1 | 0s         | sidecar   |

Anything else we need to know:

We are running the compactor with the following retention times:

--retention.resolution-raw=32d
--retention.resolution-5m=62d
--retention.resolution-1h=367d

And have enbled "--query.auto-downsampling" in querier
@bwplotka
Copy link
Member

Ack, let's take a look! Thanks for reporting.

@GiedriusS
Copy link
Member

Is it always reproducible? I wonder if #2805 is related.

@zltyfsh
Copy link
Author

zltyfsh commented Aug 20, 2020

Yes, whenever I run a query over a longer timerange the store panics. In fact both instances our of store panics at (about) the same time:

store-0:

level=debug ts=2020-08-19T07:51:57.450321718Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-1\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
panic: runtime error: slice bounds out of range [39872:35744]

goroutine 14199 [running]:
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).loadSeries(0xc0018e5c00, 0x266a460, 0xc0071da900, 0xc0067af900, 0x15, 0x20, 0x0, 0x9f7460, 0xa85b30, 0x2, ...)
        /go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1851 +0x9d5
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).PreloadSeries.func2(0x43ddb6, 0x228c3e0)
        /go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1829 +0x9d
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc003475080, 0xc006464fc0)
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:54 +0x66

store-1:

level=debug ts=2020-08-19T07:51:57.493120666Z caller=bucket.go:830 msg="Blocks source resolutions" blocks=16 MaximumResolution=720000 mint=1590044100000 maxt=1597820400000 lset="{atlas_cluster=\"stm\", replica=\"prometheus-0\"}" spans="Range: 1592136000000-1597708800000 Resolution: 300000\nRange: 1597708800000-1597816800000 Resolution: 0"
panic: runtime error: slice bounds out of range [41456:32288]

goroutine 14848 [running]:
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).loadSeries(0xc0026c9bc0, 0x266a460, 0xc002ab0000, 0xc004618b00, 0x15, 0x20, 0x100010000, 0x9f81e0, 0xa91b80, 0xc0055f3740, ...)
        /go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1851 +0x9d5
github.com/thanos-io/thanos/pkg/store.(*bucketIndexReader).PreloadSeries.func2(0x1, 0x0)
        /go/src/github.com/thanos-io/thanos/pkg/store/bucket.go:1829 +0x9d
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0065d0c00, 0xc0055f3800)
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
        /go/pkg/mod/golang.org/x/sync@v0.0.0-20200317015054-43a5402ce75a/errgroup/errgroup.go:54 +0x66

I don't know is it's relevant to the case, but both we and #2805 are using a object storage solution from Cloudian.

@bwplotka
Copy link
Member

Still relevant I guess and to be checked, help wanted.

@zltyfsh
Copy link
Author

zltyfsh commented Sep 28, 2020

We do have some other Thanos-instances using the same S3-endpoint (but another bucket) which do not panic.

I've also started removing "suspect" blocks (we had some where chunk 1 to 5 was missing) from S3, but so far the effort has been fruitless.

@MegaEle MegaEle mentioned this issue Oct 19, 2020
Closed
2 tasks
@sevagh sevagh mentioned this issue Oct 20, 2020
Closed
@stale
Copy link

stale bot commented Nov 27, 2020

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

@stale stale bot added the stale label Nov 27, 2020
@stale
Copy link

stale bot commented Dec 12, 2020

Closing for now as promised, let us know if you need this to be reopened! 🤗

@stale stale bot closed this as completed Dec 12, 2020
@zecke zecke mentioned this issue Dec 12, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug component: store difficulty: medium Hacktoberfest help wanted stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zltyfsh @GiedriusS @bwplotka