store-gateway: refactor BucketStore to have proper module lifecycle

[dimitarvdimitrov]

The BucketStore and the ReaderPool it uses need to start goroutines. It's hard to keep track of those and hard to clean them up. #8281 introduced another background goroutine. See

mimir/pkg/util/test/leak.go

Lines 27 to 30 in 36f96f1

	// The store-gateway BucketStore starts a goroutine in the index-header readers pool and
	// it gets closed when we close the BucketStore. However, we currently don't close BucketStore
	// on store-gateway termination so it never gets terminated.
	goleak.IgnoreTopFunction("github.com/grafana/mimir/pkg/storegateway/indexheader.NewReaderPool.func1"),

The store-gateway would be easier to navigate and less prone to bugs if the BucketStore has proper lifecycle functions like start, run, and stop. Like a lot of other internal services.

This issue is a follow-up for this comment thread #8281 (comment)

[dimitarvdimitrov]

related comment from @pracucci #8281 (comment)

[dimitarvdimitrov]

The changes became slightly long and hard to follow. I plan to open a few PRs for this

• store-gateway: Add services for BucketStore(s) #8551
• store-gateway: Make Snapshotter a service #8552
• store-gateway: Move eager loading to BucketStore #8602
• store-gateway: Make index reader pool a service #8603
• store-gateway: properly close LazyBinaryReader on shutdown #8610