azure v0.34.0 - gopanic: send on closed channel

[ahurtaud]

Hello,

with the latest version of thanos v0.34.0 running a compactor on an Azure Container storage. We are seeing from time to time some go panics while uploading the blocks. I think this is new to v0.34.0 and was working ok with v0.33.x

We are running in the official docker image with the following config:

args

      --log.level=debug
      --data-dir=/data
      --objstore.config=
        type: AZURE
        config:
          container: "argos-argos-amadeus-com"
          endpoint: "privatelink.blob.core.windows.net"
          storage_account: "$(OBJSTORE_ACCESS_KEY)"
          storage_account_key: "$(OBJSTORE_SECRET_KEY)"
          max_retries: 0
          http_config:
            insecure_skip_verify: true
      --wait
      --wait-interval=30m
      --compact.cleanup-interval=30m
      --compact.concurrency=8
      --downsample.concurrency=2
      --compact.blocks-fetch-concurrency=1
      --block-files-concurrency=1
      --block-meta-fetch-concurrency=5
      --web.disable
      --compact.progress-interval=10m
      --http-address=0.0.0.0:8080
      --consistency-delay=30m
      --retention.resolution-raw=35d
      --retention.resolution-5m=100d
      --retention.resolution-1h=100d
      --compact.skip-block-with-out-of-order-chunks

Go Panic stack trace

ts=2024-02-26T12:19:24.936631613Z caller=downsample.go:391 level=info msg="downsampled block" from=01HQ3G1QHTRFY7KSAJ2RJRXXT4 to=01HQJNH9DAP3BQCEKK8HT619MG duration=2m36.3914078s duration_ms=156391
ts=2024-02-26T12:19:37.839394217Z caller=azure.go:311 level=debug msg="uploading blob" blob=01HQJNH9DAP3BQCEKK8HT619MG/chunks/000001
panic: send on closed channel
goroutine 1233414 [running]:

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.staticBuffer.Put(...)

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:83

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).write(0xc0013c9400, {{0xc01d0a2000, 0x300000, 0x300000}, {0xc004e3c5a0, 0x58}, 0x300000})

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:179 +0x39a

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).sendChunk.func1()

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:153 +0xb6

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer.func1()

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:62 +0x35

created by github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer in goroutine 1233411

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:60 +0xc6

panic: send on closed channel
goroutine 1233413 [running]:

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.staticBuffer.Put(...)

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:83

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).write(0xc0013c9400, {{0xc01c5a8000, 0x300000, 0x300000}, {0xc00529a240, 0x58}, 0x300000})

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:191 +0x37c

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob.(*copier).sendChunk.func1()

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/blockblob/chunkwriting.go:153 +0xb6

github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer.func1()

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:62 +0x35

created by github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/shared.NewStaticBuffer in goroutine 1233411

/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.5.1/internal/shared/transfer_manager.go:60 +0xc6

[MichaHoffmann]

Oh, it looks like the azblob library is very old. It could be this issue; in fact the stack traces look pretty similar.

[ahurtaud]

@MichaHoffmann Hello Michael, can we please make sure Thanos updates the go modules to get this fix in the next thanos versions?
I think the dependency is not updated.
I dont have right now the project setup on my localdev, so i guess its faster for anyone of you to upgrade and create a Pull request there. If it's not that easy for you, tell me and I'll try to find some time to do it.

Thank you :)

https://github.com/thanos-io/thanos/blob/main/go.mod#L67

[yeya24]

@ahurtaud I just openend a pr to bump up the version. thanos-io/thanos#7228

Sometimes maintainers might miss updating the dependency on the main repo. When anyone notices it, feel free to upgrade the go mod on Thanos and open a pr. Leaving a comment in the issue helps as well.