[release-1.24] --egress-selector-mode=cluster|pod are broken

[brandond]

• Backport --egress-selector-mode=cluster|pod are broken #7332

[mdrahman-suse]

Validated on branch release-1.24 with commit f575bd1

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

PRETTY_NAME="Ubuntu 22.04.1 LTS"

Cluster Configuration:

1 server

Config.yaml:

# config 1
write-kubeconfig-mode: "0644"
debug: true
token: summer
cluster-init: true
egress-selector-mode: pod

# config 2
write-kubeconfig-mode: "0644"
debug: true
token: summer
cluster-init: true
egress-selector-mode: cluster

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s
3. Check logs for `Tunnel server egress proxy updating Node Pod IP

Replication Results:

• k3s version used for replication:

k3s version v1.24.13+k3s1 (3f79b289)
go version go1.19.8

• From logs, observed below
• For egress-selector-mode: pod

$ sudo journalctl -u k3s | grep "Tunnel server"
May 17 21:28:04 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:04Z" level=info msg="Tunnel server egress proxy mode: pod"
May 17 21:28:04 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:04Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
May 17 21:28:08 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:08Z" level=debug msg="Tunnel server handing HTTP/1.1 GET request for /v1-k3s/connect from 127.0.0.1:50232"
May 17 21:28:19 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:19Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 IP 172.31.4.150/32"
May 17 21:28:20 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:20Z" level=debug msg="Tunnel server handing HTTP/1.1 GET request for /v1-k3s/connect from 172.31.4.150:40580"
May 17 21:28:53 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:53Z" level=debug msg="Tunnel server handing HTTP/1.1 CONNECT request for //10.42.0.5:10250 from 127.0.0.1:40490"
May 17 21:28:53 ip-172-31-4-150 k3s[7731]: time="2023-05-17T21:28:53Z" level=debug msg="Tunnel server egress proxy dialing 10.42.0.5:10250 directly"
...

• For egress-selector-mode: cluster

$ sudo journalctl -u k3s | grep "Tunnel server"
May 17 21:57:23 ip-172-31-4-150 k3s[14479]: time="2023-05-17T21:57:23Z" level=info msg="Tunnel server egress proxy mode: cluster"
May 17 21:57:23 ip-172-31-4-150 k3s[14479]: time="2023-05-17T21:57:23Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
May 17 21:57:25 ip-172-31-4-150 k3s[14479]: time="2023-05-17T21:57:25Z" level=debug msg="Tunnel server handing HTTP/1.1 GET request for /v1-k3s/connect from 127.0.0.1:50692"
May 17 21:57:35 ip-172-31-4-150 k3s[14479]: time="2023-05-17T21:57:35Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 IP 172.31.4.150/32"
May 17 21:58:03 ip-172-31-4-150 k3s[14479]: time="2023-05-17T21:58:03Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 IP 172.31.4.150/32"
May 17 21:58:09 ip-172-31-4-150 k3s[14479]: time="2023-05-17T21:58:09Z" level=debug msg="Tunnel server handing HTTP/1.1 CONNECT request for //10.42.0.6:10250 from 127.0.0.1:46474"

Validation Results:

• k3s version used for validation:

k3s version v1.24.13+k3s-f575bd1c (f575bd1c)
go version go1.19.8

• From logs, observed below
• For egress-selector-mode: pod

$ sudo journalctl -u k3s | grep "Tunnel server"
May 17 21:42:55 ip-172-31-4-150 k3s[11059]: time="2023-05-17T21:42:55Z" level=info msg="Tunnel server egress proxy mode: pod"
May 17 21:42:56 ip-172-31-4-150 k3s[11059]: time="2023-05-17T21:42:56Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 IP 172.31.4.150/32"
May 17 21:43:01 ip-172-31-4-150 k3s[11059]: time="2023-05-17T21:43:01Z" level=debug msg="Tunnel server handing HTTP/1.1 GET request for /v1-k3s/connect from 172.31.4.150:45530"
May 17 21:43:15 ip-172-31-4-150 k3s[11059]: time="2023-05-17T21:43:15Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 Pod IP 10.42.0.3/32"
May 17 21:43:33 ip-172-31-4-150 k3s[11059]: time="2023-05-17T21:43:33Z" level=debug msg="Tunnel server handing HTTP/1.1 CONNECT request for //10.42.0.6:10250 from 127.0.0.1:53290"
May 17 21:43:33 ip-172-31-4-150 k3s[11059]: time="2023-05-17T21:43:33Z" level=debug msg="Tunnel server egress proxy dialing 10.42.0.6:10250 directly"

• For egress-selector-mode: cluster

$ sudo journalctl -u k3s | grep "Tunnel server"
May 17 22:03:04 ip-172-31-4-150 k3s[17438]: time="2023-05-17T22:03:04Z" level=info msg="Tunnel server egress proxy mode: cluster"
May 17 22:03:05 ip-172-31-4-150 k3s[17438]: time="2023-05-17T22:03:05Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 IP 172.31.4.150/32"
May 17 22:03:42 ip-172-31-4-150 k3s[17438]: time="2023-05-17T22:03:42Z" level=debug msg="Tunnel server egress proxy updating Node ip-172-31-4-150 Pod IP 10.42.0.6/32"
May 17 22:03:42 ip-172-31-4-150 k3s[17438]: time="2023-05-17T22:03:42Z" level=debug msg="Tunnel server handing HTTP/1.1 CONNECT request for //10.42.0.6:10250 from 127.0.0.1:34240"
May 17 22:03:42 ip-172-31-4-150 k3s[17438]: time="2023-05-17T22:03:42Z" level=debug msg="Tunnel server egress proxy dialing 10.42.0.6:10250 directly"