nightly 5-4-High-Availability failure: ESX host is temporarily disconnected

[AngieCris]

From portlayer log:

Dec 27 2017 22:37:52.768Z ERROR op=290.26: unexpected fault on task retry: &types.HostNotConnected{HostCommunication:types.HostCommunication{RuntimeFault:types.RuntimeFault{MethodFault:types.MethodFault{FaultCause:(*types.LocalizedMethodFault)(nil), FaultMessage:[]types.LocalizableMessage(nil)}}}}
Dec 27 2017 22:37:52.792Z DEBUG op=290.26: Unhandled fault while attempting to destroy vm fd5e455f572cbd2c0f1a07198b94ab1d96d713431a9f2858efccf15c7a02f357: &types.HostNotConnected{HostCommunication:types.HostCommunication{RuntimeFault:types.RuntimeFault{MethodFault:types.MethodFault{FaultCause:(*types.LocalizedMethodFault)(nil), FaultMessage:[]types.LocalizableMessage(nil)}}}}

5-4-High-Availability.zip

[hickeng]

Possible candidate for adding a task retry - deleting a container should be viable regardless of whether a single host is down.

[hickeng]

#6370 is discussing which task errors should be retried at a low level in the tasks package (e.g. TaskInProgress, HostNotConnected, etc) and which should propagate up to the higher level logic for re-dispatch (ConcurrentModificationError)

[hickeng]

Talking to dbeard I'm told that we could well be seeing a delay in updating the host list for routing operations, however I don't see why we should settle for this given the HA remediation has already occurred.
@mhagen-vmware derek has suggested we grab a support bundle when we see this again and open an issue. I've opened bug2057397 with the details we have and the vpxd log fragment gathered as part of the VCH log bundle. Adding this to the log collection epic as it's another case where we'd like to be able to trigger specific log collection on a given symptom.

[hickeng]

dup of #6667. Updated that with the visible symptom observed here.