[release-1.24] k3s server exits instead of retrying when receiving "too many learners" error from etcd join

[brandond]

• Backport k3s server exits instead of retrying when receiving "too many learners" error from etcd join #7000

[est-suse]

##Environment Details

Verified existing behavior using VERSION=v1.24.13+k3s1
Verified new behavior using COMMIT=f575bd1cc6ac19995e014bb5fde8bd98164dad05

Node(s) CPU architecture, OS, and version:

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Cluster Configuration:

NAME               STATUS   ROLES                       AGE     VERSION
ip-172-31-13-101   Ready    control-plane,etcd,master   6m59s   v1.24.13+k3s-f575bd1c
ip-172-31-2-145    Ready    control-plane,etcd,master   3m56s   v1.24.13+k3s-f575bd1c
ip-172-31-3-244    Ready    control-plane,etcd,master   8m16s   v1.24.13+k3s-f575bd1c
ip-172-31-6-85     Ready    <none>                      12s     v1.24.13+k3s-f575bd1c
ip-172-31-8-115    Ready    control-plane,etcd,master   2m59s   v1.24.13+k3s-f575bd1c
``

config.yaml

write-kubeconfig-mode: 644
debug: true
token: test
protect-kernel-defaults: true
cluster-init: true
``

Reproduced existing join failures

Install K3s curl https://get.k3s.io | INSTALL_K3S_COMMIT=f575bd1cc6ac19995e014bb5fde8bd98164dad05 INSTALL_K3S_SKIP_ENABLE=true sh -s - server
Start multiples nodes at the same time
Check the error logs

note on secondary nodes perform the installation with INSTALL_K3S_SKIP_ENABLE=true so that you can more easily run them quickly in tandem to trigger the race condition.

Results:

Existing too many learners error observed on existing releases.

May 16 17:33:29 ip-172-31-2-145 k3s[4759]: time="2023-05-16T17:33:29Z" level=fatal msg="ETCD join failed: etcdserver: too many learner members in cluster"
May 16 17:33:36 ip-172-31-2-145 k3s[4784]: time="2023-05-16T17:33:36Z" level=fatal msg="ETCD join failed: etcdserver: too many learner members in cluster"
May 16 17:33:43 ip-172-31-2-145 k3s[4809]: time="2023-05-16T17:33:43Z" level=fatal msg="ETCD join failed: etcdserver: too many learner members in cluster"
May 16 17:33:49 ip-172-31-2-145 k3s[4835]: time="2023-05-16T17:33:49Z" level=fatal msg="ETCD join failed: etcdserver: too many learner members in cluster"

Validation Steps

May 16 17:05:23 ip-172-31-13-101 k3s[3442]: time="2023-05-16T17:05:23Z" level=info msg="Adding member ip-172-31-13-101-dd54aa99=https://172.31.13.101:2380 to etcd cluster [ip-172-31-3-244-56a0d0bb=https://172.31.3.244:2380]"
May 16 17:05:52 ip-172-31-13-101 k3s[3473]: time="2023-05-16T17:05:52Z" level=info msg="Adding member ip-172-31-13-101-dd54aa99=https://172.31.13.101:2380 to etcd cluster [ip-172-31-3-244-56a0d0bb=https://172.31.3.244:2380]"
May 16 17:07:47 ip-172-31-13-101 k3s[3850]: time="2023-05-16T17:07:47Z" level=info msg="Adding member ip-172-31-13-101-d3e709d7=https://172.31.13.101:2380 to etcd cluster [ip-172-31-3-244-8939dd93=https://172.31.3.244:2380]"
May 16 17:22:19 ip-172-31-13-101 k3s[5387]: time="2023-05-16T17:22:19Z" level=info msg="Adding member ip-172-31-13-101-65e1ef64=https://172.31.13.101:2380 to etcd cluster [ip-172-31-3-244-db40bbca=https://172.31.3.244:2380]"