Helm : add hpa for the ingesters

[QuantumEnigmaa]

What this PR does

This PR adds a hpa resource to the ingester component in the mimir-distributed helm chart.

As the scaling down process is quite tricky as if not done properly, this could come with the loss of the data stored on the disk os the scaled down replica. To avoid that, the ingester's scaled down replica must be able to flush its data to the object storage before getting terminated. To achieve this, I looked at the PR that introduced hpa for loki components here.

I'm not sure whether the path for the added lifecycle is correct so please feel free to update it if that's the case.

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[QuantumEnigmaa]

@dimitarvdimitrov Do you have any thoughts on this ?

[dimitarvdimitrov]

thank you for opening the PR. Unfortunately, the loki ingesters and mimir ingesters don't have the same design. Because of this I don't think this simple HPA will work. The main reason is that zones need to be scaled one-by-one with some waiting period between each so that metrics are shipped to object store and are queried from there instead of ingesters.

This PR is laying the foundations for autoscaling #6733 - allowing the rollout operator to manage the lifecycle of ingester zones and sequentially scale them down