Addressing review feedback about docs for singleuser.cloudMetadata.blockWithIpTables

[consideRatio]

I merged #3179 without addressing a comment by @yuvipanda (below) since resolving #3180 would influence that specific documentation anyhow. So, this issue represents the idea to address Yuvi's comment, but after acting on #3180.

One thing I'd like added is a reference to dnsPortsCloudMetadataServer in the documentation for blockWithIptables, saying that even when it is true, DNS access is allowed and controlled by dnsPortsCloudMetadataServer. With that, this looks good to me.

The gist I'm putting it off to document this quickly in #3179 is because I'm not confident on the general truth of the because: I don't understand iptables that well, and partially because the network policy enforcement can be done in different ways and I'm not sure about the consequences of that. For example Calico use iptables (apparently prepended rules rather than appended rules by default), and Cilium use eBPF to manage networking permissions etc. How does all of this couple?

[manics]

I can't think of a technical reason why blockWithIptables: true is needed when egress network policies are functional- is there a reason to use Cilium but not networkpolicies? Can we simplify things by saying blockWithIptables: true is not supported when networkpolicies are also used for blocking traffic? It sounds like it could end up being implementation dependent.

[consideRatio]

It sounds like it could end up being implementation dependent.

Yes, or at least I dare not say its not.

I suggest a resolution to this issue and #3180 in the PR #3185, where I let the ambiguous configuration blocking and allowing access cause a failure during template rendering, and where I document that blocking access with the iptables is only blocking TCP port 80, making there only be one ambiguous config to fail on.

[consideRatio]

Closed by #3192