Installation on GKE (standard & autopilot) fails with Dataplane V2 & kube-dns(Cilium) #3167
Comments
|
I investigated this further and found a work around. However, the core network policies may need to be reviewed for Dataplane V2. I have described the issue and troubleshooting steps with a solution at the link below, |
|
Excellent debugging and tracking what the core issue was @vizeit!! Okay, so with dagaplane v2 you needed to add rules to add egress rules to communicate with the dns server - because the default rules failed to do so. Your rules looked like: networkPolicy:
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53Nice!!! I learned about the ability to target a namespace name based on the special label metadata.name. I think we can improve our defaults in this helm chart based on this because allowing access to the dns server by default has been a bit tricky. Thank you for your investigative efforts and detailed writeup, i learned about what gke dataplane v2 is and a debugging trick for it! |
|
Happy to contribute to the community! |
|
This chart provides a egress rule by default, configurable via The rule looks like below, and is defined here: # Allow outbound connections to the DNS port in the private IP ranges
- ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
to:
- ipBlock:
cidr: 10.0.0.0/8
- ipBlock:
cidr: 172.16.0.0/12
- ipBlock:
cidr: 192.168.0.0/16This rule was designed to arrive at a DNS server without knowing that much about it, and not allowing more than needed. But it is not doing the work for some reason on dataplane v2 as you had to add another rule targetting the pods in the kube-system namespace instead (something I didn't know was possible until now). It seems that in k8s 1.22, we can assume the new label as documented here. A key question in my mind is why the rules we had didn't work. Is it because the - ports:
- protocol: UDP
port: 53
to:
- ipBlock:
cidr: 10.0.0.0/8 # this subnet range includes 10.114.1.72I've tried to look into source / destination differences via docs and reference. The reference sais:
Due to this, I'm confused. Why doesn't the z2jh default rule provided work? Is this a bug in dataplane v2 rather? If so, its important to know that what GKE calls dataplane v2 is really simply Cilium, so it could be either GKE's dataplane v2 specifics related to deploying cilium, or cilium itself. |
|
@vizeit I think the networking rule z2jh provides by default should work, and its a bug in Cilium described here that causes the issue: cilium/cilium#25998. If you want to confirm this, I think you could use your test cluster and do This is because the bug describes that cilium is failing specifically when multiple ipBlock.cidr are specified as we have, and that is the key difference between the rule you added and the rule we provided with the z2jh chart I think. |
consideRatio
changed the title
|
In my understanding Dataplane V2 uses eBUF instead of iptables and that may be the reason existing rule will not work |
I will try |
|
I think the Helm chart should add a new rule by default like the one you defined because its more narrow, targetting pods in the k8s namespace kube-system specifically instead of all pods. Also like that, we would have a workaround for the cilium bug by default. Followup actionsI think the following makes sense to followup this issue with. I'd like to be involved working the last three in some way, I want to ensure that these changes that are security related and breaking are implemented quickly and communicated clearly for the 3.0.0 release. If you want to work them as well @vizeit you are most welcome to just go for it. I'm vacationing and will work more the week after the one coming up now.
|
I tested as per your comment. The hub pod remained in the reported error condition, the reported issue did not resolve with the suggested change. Please let me know if you have any questions Steps followed:
|
|
Did you delete the pod after changing the network policy, so we know its not caused by a failure to re-apply the rule or similar? |
I did check again by deleting the pod and the same error remains, although it does not require to delete the pod to check that updated policy is applied or not. Updated policy gets applied right away. You can double check if you want |
|
Hmmmm, so then it maybe isnt the same bug, but another bug? I think the network policy as defined by this chart isnt enforced correctly by cilium. |
In my understanding, it is not a cilium bug. With cilium, the network policy for anything outside the cluster is using IP. The cilium bug you have referred here will apply to IP block for outside the cluster IPs. The kube-system is within the cluster and cannot be enforced with an IP |
I don't understand you well here. I'm thinking that what NetworkPolicy specifies should be enforced the same way no matter what network policy controller does the job - according to a kubernetes official definition of how it should be enforced. I understand it as a NetworkPolicy resource defining an egress rule like below should allow opening connections to port 53 destinations on an IP like 10.x.y.z (because 10.0.0.0/8). - ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
to:
- ipBlock:
cidr: 10.0.0.0/8I do believe that you can allow egress to a k8s service and its exposed endpoints on pods based on IPs like this, otherwise we would have had this failure in other k8s clusters where a k8s pod called |
|
Looking at cilium docs, it seems there are a lot of notes about CIDR and IPAM (ip address management), so I suspect this is perhaps an expected behaviour of cilium being limited by GKE to not support this or similar: https://docs.cilium.io/en/latest/network/concepts/ipam/ |
|
Do you know what IPAM configuration cilium has on GKE? I don't understand much what I'm asking now, but I think this is the thread worth digging into to understand why what we have currently doesn't work - but the path onwards remains the same I think: switch to another rule based UDP/IP port 53 in the |
|
IPAM on GKE is kubernetes. I have described detailed steps in my post below to find the root cause of the issue. The issue is not limited to Dataplane V2, any Kubernetes cluster with Cilium will have the issue. This DNS behavior is well-known and it is based on the way it is implemented, as I indicated in my previous comments. I hope this post helps to get more understanding of the issue |
|
I tracked this down to be a known limitation with Cilium, and has been for a long time. This documentation mentions it as a known missing feature, referencing cilium/cilium#9209 as the issue to track. |
|
Please refer to K8 documentation at https://kubernetes.io/docs/concepts/services-networking/network-policies/
|
vizeit
changed the title
|
While working on my implementation, I have found that this issue will be limited to the GKE Autopilot & Standard with Dataplane-V2 & kube-dns for the Google managed Kubernetes. Google is moving GKE to Cloud DNS which will be external to GKE clusters.
singleuser:
cloudMetadata:
blockWithIptables: false
@consideRatio not sure if you want to track this cloud DNS issue separately, I can open a new issue |
|
I opened #3179 with a proposal on how to address various situations including when Cloud DNS is reached at the non-private/non-pod-cidr IP of the cloud metadata server. Thank you soo much for digging into the details about this, I think I've finally understood this well enough to document it. |
|
Do you think #3179 does the trick? With it, I think the only thing needed for use with Cilium or GKE's Cloud DNS could be: singleuser:
cloudMetadata:
blockWithIptables: false |
3179 will partially address the Cloud DNS issue for GKE. The single user core network policy will need to allow egress to the GCP meta server 169.254.169.254. Otherwise single user pod will fail to launch. |
Happy to help! Thank you for looking into the issue |
Yepp! This is actually part of #3179, all network policy resources defined by the jupyterhub chart get the same "egressAllowRules" configuration, so both singleuser.networkPolicy.egressAllowRules.dnsPortsKubeSystemNamespace and singleuser.networkPolicy.egressAllowRules.dnsPortsCloudMetadataServer is available and true by default with #3179. |
Bug description
This is the continuation of the ticket 3163. It was closed with the comment to search on https://discourse.jupyter.org for similar issues. This issue reported here did not match with any existing discussions on the discourse for error 599.
The issue is that when GKE cluster, either standard or Autopilot, is created with Dataplane V2, the hub fails to start with the following error,
Expected behaviour
The installation should complete successfully
Actual behaviour
The installation fails with hub pod in crashbackoff state. The logs show the following error message,
api_request to the proxy failed with status code 599, retrying...
How to reproduce
GKE Autopilot:
config.yaml content:
GKE standard:
config.yaml content:
Your personal set up
Full environment
Configuration
# jupyterhub_config.pyLogs
The text was updated successfully, but these errors were encountered: