Do we need to remove node on failure?

[JackyChiu]

From #15 (comment) we saw that changes to the etcd cluster won't be allowed if it isn't healthy. The case that we are concerned about is if a node in the cluster gets taken down. We think it might jeopardize the health of the cluster and not allow changes.

If we do need to remove it on failure, we need to build a mechanism for it.

[bkzhang]

MemberRemove to remove, and to keep track of which nodes are healthy, a possible solution is using cli.Status and checking StatusResponse.Errors. Another possibility is to see if a node's changed using watcher however we need to peerURL to remove member so this may not be possible

[JackyChiu]

I have just realized that this is only an issue if the cluster loses quorum. Ie. in the scenario where we have 3 nodes and one goes down, the addition of a new member will still work. Therefore it is not mandatory to explicitly remove a node on failure.

Will confirm this with a test. If this is true, I think we can ignore this issue.

EDIT: nope, looks like this is an issue: 23871c3

[bkzhang]

https://github.com/etcd-io/etcd/blob/master/Documentation/faq.md#should-i-add-a-member-before-removing-an-unhealthy-member

[JackyChiu]

https://github.com/etcd-io/etcd/blob/master/Documentation/faq.md#should-i-add-a-member-before-removing-an-unhealthy-member

This might be the reason why my tests don't work. Since when I try to add node 4 in a 3 node cluster with 1 failing, it requires 3 nodes of quorum.

[JackyChiu]

Moving this back to TODO to prioritze the project's poster fair slides and final paper.