Missing xt_recent kernel module resulting in inability to run Kubernetes with Session Affinity

[valeneiko]

Windows Build Number

Microsoft Windows [Version 10.0.19042.1055]

WSL Version

• WSL 2
• WSL 1

Kernel Version

No response

Distro Version

Ubuntu 20.04

Other Software

Kind, v0.11.1
Kubernetes, v1.20.7
Docker Desktop (Windows), version 3.4.0

Repro Steps

1. Install Docker Desktop (link)
2. Get Kind (link)

   go get sigs.k8s.io/kind@v0.11.1

3. Create a Kubernetes cluster

   kind create cluster

4. Create a Service in Kubernetes with sessionAffinity: ClientIP (kubectl apply -f service.yaml).

   # service.yaml
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: session-affinity-test
   spec:
     selector:
       matchLabels:
         app: session-affinity-test
     template:
       metadata:
         labels:
           app: session-affinity-test
       spec:
         containers:
         - name: session-affinity-test
           image: mendhak/http-https-echo
           resources:
             limits:
               memory: "128Mi"
               cpu: "500m"
           ports:
           - containerPort: 8080
             name: web
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: session-affinity-service
     namespace: default
   spec:
     ports:
     - name: web
       port: 9093
       targetPort: web
     selector:
       app: session-affinity-test
     sessionAffinity: ClientIP    

5. Try to access the service, or view Kube Proxy logs

Expected Behavior

Kube Proxy should have successfully applied iptables rules and Kubernetes service is accessible.

Some discussion about this issue: kubernetes-sigs/kind#1740.

Yes, it looks like the current WSL2 Kernel is built without xt_recent, needed by iptables -m recent ... which kube-proxy uses to implement sessionAffinity: ClientIP. Custom Kernel built with CONFIG_NETFILTER_XT_MATCH_RECENT=y fixed it for me. Submitted microsoft/WSL2-Linux-Kernel#198 (4.19.y) and microsoft/WSL2-Linux-Kernel#199 (5.4.y)

Actual Behavior

Service is inaccessible and any other service created afterwards is inaccessible. Which is caused by inability to apply iptables rules.

The following log is present in Kube Proxy:

E0720 14:29:10.934607       1 proxier.go:1507] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.8.3 (legacy): Couldn't load match `recent':No such file or directory

Error occurred at line: 96
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
)
I0720 14:29:10.934636       1 proxier.go:779] Sync failed; retrying in 30s

Diagnostic Logs

No response

[WSLUser]

I've updated my link in the issue mentioned. I had a PR but MS decided that despite kernel being OSS, they won't accept outside contributions.

[therealkenc]

=m on native

[jasper-d]

@craigloewen-msft Any news on this? It's been half a year. PRs that addressed this issue were opened over at microsoft/WSL2-Linux-Kernel and closed without a good reason.

[douglascamata]

Any news on this? Having to recompile the WSL2 kernel to have a good working environment for k8s is annoying and consumes a good amount of time unless you have a powerful machine. It would be great to include this in the default build.

[damarshall]

MS team - without this fix there's a whole class of WSL2 use-cases (developer local K8s on WSL2) that are invalidated.
Please add this feature in an upcoming build - WSL2 has great potential as a development environment, but needs to be able to run local K8s without issue.

[thesalmonidae]

This should be fixed as soon as possible.

@craigloewen-msft any news?

[lk-1984]

I would also like to get this fixed instead of having to build own version of the kernel.

[8ar10der]

Vote! xt_recent kernel module can help all the developer who doing some work on K8s in WSL2

[Rainson12]

this issue has caused me nightmares since the whole cluster wasnt able to communicate anymore just because one container has sessionAffinity: ClientIp set.

[douglascamata]

@jiayali-ms you closed both PRs to the kernel trying to implement this fix (microsoft/WSL2-Linux-Kernel#199 and microsoft/WSL2-Linux-Kernel#198). On your closing comment, you said:

While we understand that it is frustrating to have a PR closed, it is never our intention to ignore feedback or stifle community discussions. We are taking this opportunity to increase transparency on how we work with the WSL community and ensure all members are routed to the appropriate resources.

If you would like contribute to, discuss, or report an issue relating to WSL, or the WSL 2 Linux kernel configuration, please do so at the WSL GitHub. The team is actively monitoring and responding to user feedback there: https://github.com/microsoft/WSL.

We are here, reporting the issue that causes so many k8s environments to be broken in WSL 2 and yet we didn't hear anything back.

So, what's up? Is this going to be fixed?

Recompiling the WSL 2 kernel only for this reason is an awful solution.

[douglascamata]

@therealkenc do you happen to know anything about this issue?

[alex-guo-github]

@therealkenc do you happen to know anything about this issue?

Oh, men, I am trying to provision an RKE2 worker node using Calico on Win11 WSL2 with a bridged network. And the kube-proxy pod suffers from this no xt_recent iptables failure. I am already using a custom kernel (built using native Ubuntu 22.04 and a modified ms kconfig) with many xt options enabled to support wireguard. I wonder why MS is so hostile to the Linux net stack... All they are trying to do is to limit the role of wsl a client behind NAT... besides, I still have several machines not compatible with win11, and need to figure out how to overcome the hyper-v switch nat mode...

[douglascamata]

@alex-guo-github not sure whether it will help you, but for Cilium I had to add CONFIG_NETFILTER_XT_TARGET_CT=y and CONFIG_NETFILTER_XT_TARGET_TPROXY=y too.

[davidgemel]

Don't know if anyone needs this but I created a little helm snippet to detect if the node is docker-desktop running in wsl2 so I could just not set session affinity and/or throw an error if any multiple replicas scenarios were applied on wsl2.

{{- $isWSL2 := false }}
{{ $nodes := (lookup "v1" "Node" "" "" ) }}
{{ if $nodes }}
{{ range $index, $node := $nodes.items }}
{{ if $node }}
{{ if $node.status }}
{{ if $node.status.nodeInfo }}
{{ if $node.status.nodeInfo.kernelVersion }}
{{ if ( hasSuffix "WSL2" $node.status.nodeInfo.kernelVersion ) }}
{{- $isWSL2 = true }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}
{{ end }}

[razlani]

Also voting here for incl xt_recent - recompiling linux kernels is fine but looks like folks already tried to introduce solution pr's - seemingly without unwanted side-effects!

[microsoft-github-policy-service]

This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

Thank you!

[benzman81]

this is still an issue

[mostafa]

For future reference:

The following doc article about running Kind on WSL2 explains how to build your kernel to support session affinity:
https://kind.sigs.k8s.io/docs/user/using-wsl2/#kubernetes-service-with-session-affinity