[kube-prometheus-stack] thanos-ruler can no longer communicate with Alertmanager

[TomHellier]

Describe the bug a clear and concise description of what the bug is.

It appears that since the update of alertmanager to 0.27.0 by @DrFaust92, a change to alertmanager has deprecated the /v1/ api.

It looks like the default behaviour will be updated in the next release of thanos-rule.

Alerts generated by the thanos-ruler no longer get propagated to the alertmanager instance.

ts=2024-03-22T12:16:26.773542094Z caller=alert.go:326 level=warn component=rules msg="sending alerts failed" err="bad response status 410 Gone from \"http://monitoring-alertmanager.monitoring.svc.cluster.local:9093/alertmanager/api/v1/alerts\""

What's your helm version?

argocd v2.10.3

What's your kubectl version?

1.28

Which chart?

kube-prometheus-stack

What's the chart version?

57.0.3

What happened?

Alertmanager has deprecated the v1 api, and no longer accepts messages from thanos-ruler

What you expected to happen?

alerting continued to work

How to reproduce it?

Using the kube prometheus stack, ensure thanos-ruler is alerting, and observe the error logs in thanos-ruler, and an empty alertmanager

Enter the changed values of values.yaml?

NONE

Enter the command that you execute and failing/misfunctioning.

None

Anything else we need to know?

No response

[DrFaust92]

Im not sure anything needs to be done on the chart side, you can either pin AM version to 0.26.0 or change AM config for API as referenced in the ticket