[Kubernetes] Peers ip caching causes all clusters to degrade over time #2250
Comments
grzesuav
changed the title
|
Can you elaborate about your setup? Which names do you use for |
|
hi @simonpasquier , and yes, fresh after hard reset of all instances in the same time it starts with three peers. But previously it has cluster formed along with all other alertmanager installations in our cluster (over 50 instances I guess ?) So, from what you are saying, if during those 6 hours, ip gets re-assigned to completely different pod, AM will try to form cluster with this new pod ? |
yes |
|
BTW you can set the |
|
Actually I try to resolve it with NetworkPolicies, however it is still workaround. I would still consider this as something which should be handled or at least mention in documentation (we are using prometheus operator so relay a bit on defaults there) as in our case silencing alert on one alertmanager caused somebody else missed notification. |
|
I've found this issue when trying to figure out why alertmanagers formed a mesh over cluster boundaries. I was investigating because of alerts about alertmanager being in inconsistent state fired, and nothing more serious happened. I do find it surprising that alertmanager (often used in dynamic envs like k8s) will not try to resolve the names on every connection attempt |
That's a reasonable suggestion but prometheus operator does not let you pass additional parameters to the alertmanager instance. But all in all, this is something that needs to be addressed there of course. |
In a high-dynamic environment like kubernetes, it's possible that alertmanager pods come and go on frequent intervals. The default timeout value of 6h is not suitable in that case as alertmanager will keep trying to reconnect to a non-existing pod over and over until it gives up and go through another DNS resolution process. As such, it's best to use a lower value which will allow the alertmanager cluster to recover in case of an update/rollout/etc process in the kubernetes cluster. Related: prometheus/alertmanager#2250
In a high-dynamic environment like kubernetes, it's possible that alertmanager pods come and go on frequent intervals. The default timeout value of 6h is not suitable in that case as alertmanager will keep trying to reconnect to a non-existing pod over and over until it gives up and go through another DNS resolution process. As such, it's best to use a lower value which will allow the alertmanager cluster to recover in case of an update/rollout/etc process in the kubernetes cluster. Related: prometheus/alertmanager#2250
In a high-dynamic environment like kubernetes, it's possible that alertmanager pods come and go on frequent intervals. The default timeout value of 6h is not suitable in that case as alertmanager will keep trying to reconnect to a non-existing pod over and over until it gives up and goes through another DNS resolution process. As such, it's best to use a lower value which will allow the alertmanager cluster to recover in case of an update/rollout/etc process in the kubernetes cluster. Related: prometheus/alertmanager#2250
In a high-dynamic environment like kubernetes, it's possible that alertmanager pods come and go on frequent intervals. The default timeout value of 6h is not suitable in that case as alertmanager will keep trying to reconnect to a non-existing pod over and over until it gives up and goes through another DNS resolution process. As such, it's best to use a lower value which will allow the alertmanager cluster to recover in case of an update/rollout/etc process in the kubernetes cluster. Related: prometheus/alertmanager#2250
Alertmanager in cluster mode resolves the DNS name of each peer and caches its IP address which uses on regular intervals to 'refresh' the connection. In high-dynamic environment like kubernetes, it's possible that alertmanager pods come and go on frequent intervals. The default timeout value of 6h is not suitable in that case as alertmanager will keep trying to reconnect to a non-existing pod over and over until it gives up and remove that peer from the member list. During this period of time, the cluster is reported to be in a degraded state due to the missing member. As such, it's best to use a lower value which will allow the alertmanager to remove the pod from the list of peers soon after it disappears. Related: prometheus/alertmanager#2250
Alertmanager in cluster mode resolves the DNS name of each peer and caches its IP address which uses on regular intervals to 'refresh' the connection. In high-dynamic environment like kubernetes, it's possible that alertmanager pods come and go on frequent intervals. The default timeout value of 6h is not suitable in that case as alertmanager will keep trying to reconnect to a non-existing pod over and over until it gives up and remove that peer from the member list. During this period of time, the cluster is reported to be in a degraded state due to the missing member. As such, it's best to use a lower value which will allow the alertmanager to remove the pod from the list of peers soon after it disappears. Related: prometheus/alertmanager#2250
|
I've got the same issue and able to reproduce. I think it is nature of gossip protocol which should be suppressed a bit by alertmanager since it has knowledge of peers from config file and can verify the table of available peers. UPD
You can see here is only three peers. Before making them to switch IPs there IP assignment was:
After restart few Pods few times I can make them to reuse IPs:
Since other Pods were not restarted, they still keep old IPs in their gossip available peers table. Therefore two cluster will merge into one: |
|
I am not familiar of |
|
of course, at Alertmanager level it would be an cli-argument, which people not using prom-op would need to set, otherwise some default would be used |
|
Seems there is coming TLS support for gossip in am (if not yet released). Which is one way to avoid the issue: Also some notes: thanks to @simonpasquier with sharing this docs over IRC : ) |
|
There is also one possible solution is to add cluster id: |
This is an alternate mechanism for isolating Alertmanager clusters without having to set up the right components of TLS. It should solve issues such as <prometheus#2250>, although enabling this feature will lead to loss of non-persisted state. (For example, if you rely on alertmanager cluster peering to maintain silences instead of using persistent volume storage in Kubernetes.) The Gossip label serves as the "cluster ID" idea mentioned in <prometheus#2250 (comment)>. You can enable with the command-line flag, `--cluster.gossip-label`; any non-empty string will form an effective namespace for gossip communication. If you use Prometheus Operator, you can set the `ALERTMANAGER_CLUSTER_GOSSIP_LABEL` environment variable (as Prometheus Operator does not have a way of adding additional command-line flags). You would need to modify your Alertmanager object something like: ``` kind: Alertmanager ... spec: ... containers: - name: alertmanager env: - name: ALERTMANAGER_CLUSTER_GOSSIP_LABEL value: infrastructure-eu-west-2 ... ``` This is low-security mechanism, suitable for use with Alertmanager configuration where anyone can add or remove a silence. It protects against surprising cluster expansion due to IP:port re-use.
This is an alternate mechanism for isolating Alertmanager clusters without having to set up the right components of TLS. It should solve issues such as <prometheus#2250>, although enabling this feature will lead to loss of non-persisted state. (For example, if you rely on alertmanager cluster peering to maintain silences instead of using persistent volume storage in Kubernetes.) The Gossip label serves as the "cluster ID" idea mentioned in <prometheus#2250 (comment)>. You can enable with the command-line flag, `--cluster.gossip-label`; any non-empty string will form an effective namespace for gossip communication. If you use Prometheus Operator, you can set the `ALERTMANAGER_CLUSTER_GOSSIP_LABEL` environment variable (as Prometheus Operator does not have a way of adding additional command-line flags). You would need to modify your Alertmanager object something like: ``` kind: Alertmanager ... spec: ... containers: - name: alertmanager env: - name: ALERTMANAGER_CLUSTER_GOSSIP_LABEL value: infrastructure-eu-west-2 ... ``` This is low-security mechanism, suitable for use with Alertmanager configuration where anyone can add or remove a silence. It protects against surprising cluster expansion due to IP:port re-use. Signed-off-by: Graham Reed <greed@7deadly.org>
|
this should be fixed by #3354 which allows to define a label identifying the cluster and preventing external instances to join the cluster if they don't share the same label. |
What did you do?
We have deployed several different
Alertmanager's - in different namespacesWhat did you expect to see?
Each alertmanager installation forms a separate cluster
What did you see instead? Under which circumstances?
After some time, we noticed that all those installation form one big cluster.
We think that :
--cluster.peer=DNS addressis resolved to IP address and later on, this ip address is usedEnvironment
Kubernetes cluster
System information:
will add if needed
Alertmanager version:
insert output of
alertmanager --versionhereDifferent versions - 020, 0.17, 0.18
Prometheus version:
not related
Alertmanager configuration file:
will add if needed
will add if needed
will add if needed
The text was updated successfully, but these errors were encountered: