firewall status delayed on vcenter

[andrewtchin]

In a vCenter environment, after updating the firewall allowed IP rules on one of the managed hosts the data about the updated firewall rules is delayed in updating for an undetermined amount of time.

The result of this is that vicadmin can show an incorrect firewall status and that VCH create checks will operate based on stale state and incorrectly report success/failure.

This log shows that the VCH deployed at 10.161.23.216 is allowed by the nfsClient firewall rule, even though that specific allowed IP was removed several minutes ago. The firewall is behaving correctly in this case because I can't run/attach a container with the VCH IP not in the allowed IP list.

vicadmin.log

time=2016-11-11T23:10:44.930829084Z level=debug msg=[ END ] [github.com/vmware/vic/lib/install/validate.CreateFromVCHConfig:61] [477.074µs]  
time="2016-11-11T23:10:44Z" level=info msg="Using management IP {10.161.23.216 ffffe000} for firewall check" 
time="2016-11-11T23:10:44Z" level=info msg="Firewall status: ENABLED on \"/ha-datacenter/host/10.161.22.8/10.161.22.8\"" 
time="2016-11-11T23:10:44Z" level=debug msg="filtered rules: [{{} nfsClient NFS Client false [{{} 0 65535 outbound dst tcp}]  true 0xc4205d1bc0}]" 
time="2016-11-11T23:10:44Z" level=debug msg="filtered IPs: [10.162.203.56 10.166.2.116 10.162.202.248 10.162.202.209 10.161.23.216] networks: [] allIP: false rule: nfsClient" 
time="2016-11-11T23:10:44Z" level=info msg="Firewall configuration OK on hosts:" 
time="2016-11-11T23:10:44Z" level=info msg="\t\"/ha-datacenter/host/10.161.22.8/10.161.22.8\"" 

This log shows (incorrectly) that host 10.161.23.216 is allowed by firewall rules

vic-machine create

DEBU[2016-11-11T17:25:03-06:00] Network configuration:                       
DEBU[2016-11-11T17:25:03-06:00] 	Network: client NetworkEndpoint: &{{  client } true 10.161.23.216/24 {<nil> <nil>} {{ Network:network-18 client }  {10.161.23.1 ffffff00} false [] [] []} []} 
DEBU[2016-11-11T17:25:03-06:00] 	Network: management NetworkEndpoint: &{{   } true 10.161.23.216/24 {<nil> <nil>} {{ Network:network-18 management }  {10.161.23.1 ffffff00} false [] [] []} []} 
DEBU[2016-11-11T17:25:03-06:00] 	Network: external NetworkEndpoint: &{{  external } true 10.161.23.216/24 {<nil> <nil>} {{ Network:network-18 external }  {10.161.23.1 ffffff00} true [] [] []} []} 
time=2016-11-11T17:25:03.460107249-06:00 level=debug msg=[BEGIN] 
...
[github.com/vmware/vic/lib/install/validate.(*Validator).CheckFirewall:86] 
DEBU[2016-11-11T17:25:04-06:00] Checking firewall with management network IP {10.161.23.216 ffffff00} 
INFO[2016-11-11T17:25:05-06:00] Firewall status: ENABLED on "/ha-datacenter/host/10.161.22.8/10.161.22.8" 
DEBU[2016-11-11T17:25:06-06:00] filtered rules: [{{} nfsClient NFS Client false [{{} 0 65535 outbound dst tcp}]  true 0xc420010640}] 
DEBU[2016-11-11T17:25:06-06:00] filtered IPs: [10.162.203.56 10.166.2.116 10.162.202.248 10.162.202.209 10.161.23.216] networks: [] allIP: false rule: nfsClient 
INFO[2016-11-11T17:25:06-06:00] Firewall configuration OK on hosts:          
INFO[2016-11-11T17:25:06-06:00] 	"/ha-datacenter/host/10.161.22.8/10.161.22.8" 

[stuclem]

Added the following to the Known Issues for 0.8 RC2:

---

• Firewall status delayed on vCenter Server. #3139
  If you update the firewall rules on an ESXi host to allow access from specific IP addresses, and if that host is managed by vCenter Server, there might be a delay before vCenter Server takes the updated firewall rule into account. In this case, vCenter Server continues to use the old configuration for an indeterminate amount of time after you have made the update. vic-machine create can successfully deploy a VCH with an address that you have blocked, or else fail when you deploy a VCH with an address that you have permitted.

  Workaround: Wait a few minutes and run vic-machine create again.

---

@andrewtchin is this OK? Thanks!

[andrewtchin]

Yes that's good 👍

[emlin]

Looks like this is unavoidable, besides of a document, what else should be done for it?

[andrewtchin]

I'm not sure - this issue was meant to go research and see if there are any other possible causes. At first glance it appears we query the firewall status properly, so there might not be anything to do.
Anyway it's low priority - I'm OK with closing as won't fix or leaving it open until someday since we have a doc warning

[stuclem]

If there's nothing that can be done about this and the behaviour will never change, then the doc needs to move out of the release notes and into a troubleshooting topic in the main doc. If you approve, @andrewtchin and @emlin, we can convert this issue into a doc issue.

[emlin]

I'm fine with it

[andrewtchin]

Me too

[andrewtchin]

@stuclem i added kind/docs let me know if you need anything else on this or if you end up making a separate issue we can close this one

[stuclem]

People seem happy with the fact that this is in the release notes, so closing this one.