-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
race condition when flock to chef pause file fails on a node #34
Comments
This is blocked due to @vuntz missing a profile - so he cannot log in to the system and investigate |
For the record, the "no route to host" error occurred at 2017-04-19T13:42:25.501410 (in case others need to look at production.log). |
So, I'm seeing some nodes where the nova entries for haproxy got dropped during the periodic chef-client run at 2017-04-19T14:45:17+00:00 and 2017-04-19T15:13:53+00:00. What's interesting is that it's the first periodic chef-client run after the apply failure. |
So here's what happened:
Not sure yet how to best fix it. |
Until now, we saved the role for the applied proposal early in apply_role. However, apply_role could fail after that but still before apply_role_pre_chef_client, resulting in the role containing incomplete data (since apply_role_pre_chef_client is used to process the role and add some info, like "use HA"). In case of such a failure (and that can happen when we fail to get a lock for a node), then a periodic chef-client run would use a role with incomplete information, possibly leading to changing the config on the node wrongly. Closes sap-oc/crowbar-openstack#34
I think crowbar/crowbar-core#1225 is the right fix. |
Testing the change locally. |
Steps to reproduce:
In case the node got fenced, you can recover it with:
|
Great! That's exactly how it can be reproduced. Thanks @matelakat ! |
Applying the fix:
|
Until now, we saved the role for the applied proposal early in apply_role. However, apply_role could fail after that but still before apply_role_pre_chef_client, resulting in the role containing incomplete data (since apply_role_pre_chef_client is used to process the role and add some info, like "use HA"). In case of such a failure (and that can happen when we fail to get a lock for a node), then a periodic chef-client run would use a role with incomplete information, possibly leading to changing the config on the node wrongly. This partially addresses https://bugzilla.suse.com/show_bug.cgi?id=857375 Closes sap-oc/crowbar-openstack#34
Until now, we saved the role for the applied proposal early in apply_role. However, apply_role could fail after that but still before apply_role_pre_chef_client, resulting in the role containing incomplete data (since apply_role_pre_chef_client is used to process the role and add some info, like "use HA"). In case of such a failure (and that can happen when we fail to get a lock for a node), then a periodic chef-client run would use a role with incomplete information, possibly leading to changing the config on the node wrongly. This partially addresses https://bugzilla.suse.com/show_bug.cgi?id=857375 Closes sap-oc/crowbar-openstack#34 (cherry picked from commit 23dbcca)
Backport at sap-oc/crowbar-core#29 |
Until now, we saved the role for the applied proposal early in apply_role. However, apply_role could fail after that but still before apply_role_pre_chef_client, resulting in the role containing incomplete data (since apply_role_pre_chef_client is used to process the role and add some info, like "use HA"). In case of such a failure (and that can happen when we fail to get a lock for a node), then a periodic chef-client run would use a role with incomplete information, possibly leading to changing the config on the node wrongly. This partially addresses https://bugzilla.suse.com/show_bug.cgi?id=857375 Closes sap-oc/crowbar-openstack#34 (cherry picked from commit 23dbcca)
We had an outage in production
Symptoms:
nova api no longer available. haproxy.cfg was changed by local chef-client run on the control-node running haproxy. All nova services where removed from haproxy.cfg. nova.conf was changed as well: all listen IPs/ports were reset to default.
To fix it quick we restored the local chef backup of the config files.
First analysis:
Nova proposal was applied to include new computes. One older compute had an issue and the proposal failed with:
In the same time (or right after) the local/periodic chef-client run on the control-nodes happened (because the flock didn't ran over all nodes) and change the nova.conf and the haproxy.cfg.
The local chef-client run failed:
The node mentioned in the proposal error was no longer in the kvm compute list of the nova proposal, but it still had the nova role assigned on its crowbar role. We manually removed it and re-applied nova proposal. That run was successful. No haproxy.cfg or nova.conf changes. The following local chef-client runs were also sucessful.
So my guess is, that the missing flock on the chef pause file, resulted in this race condition, where the local chef-client got wrong/default data (from the founder?).
The text was updated successfully, but these errors were encountered: