Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.26] - API complains on resources exist during helm install and which actually not #8841

Closed
brandond opened this issue Nov 14, 2023 · 1 comment
Closed

[Release-1.26] - API complains on resources exist during helm install and which actually not #8841

brandond opened this issue Nov 14, 2023 · 1 comment
Assignees
@brandond @VestigeJ

Comments

@brandond
Copy link
Member

Backport fix for API complains on resources exist during helm install and which actually not
@brandond brandond self-assigned this Nov 14, 2023
@brandond brandond mentioned this issue Nov 16, 2023
Merged
@VestigeJ
Copy link

##Environment Details
Validated using VERSION=v1.26.11-rc1+k3s1

Infrastructure

  • Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME            STATUS   ROLES                       AGE   VERSION
ip-3-17-16-17   Ready    control-plane,etcd,master   36m   v1.26.11-rc1+k3s1
ip-3-14-15-15   Ready    control-plane,etcd,master   37m   v1.26.11-rc1+k3s1
ip-3-13-13-33   Ready    <none>                      35m   v1.26.11-rc1+k3s1
ip-3-15-15-12   Ready    control-plane,etcd,master   35m   v1.26.11-rc1+k3s1

Config.yaml:

server: https://3.14.15.15:6443
write-kubeconfig-mode: 644
debug: true
token: YOUR_TOKEN_HERE
profile: cis
selinux: true
kubelet-arg:
  - max-pods=250
kube-controller-manager-arg:
  - node-cidr-mask-size=22
node-external-ip: 3.17.16.17

Validation Steps

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf ~/90-kubelet.conf
$ sudo cp 60-rke2-cis.conf 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_K3S_VERSION=v1.26.11-rc1+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh
$ set_kubefig
$ kgn
$ kgp -A
$ wget https://gist.githubusercontent.com/brandond/595f80751b037e945f15ba64d3c798e9/raw/da5b48646e4560ddc94170704f6bb34663bd6ed8/nginx.yaml
$ wget https://gist.githubusercontent.com/brandond/595f80751b037e945f15ba64d3c798e9/raw/da5b48646e4560ddc94170704f6bb34663bd6ed8/observer.yaml
$ k create -f nginx.yaml -f observer.yaml
$ kg deploy -A
$ k scale deploy observer -n test --replicas=205
$ sudo journalctl -u k3s
$ k scale deploy observer -n test --replicas=5
$ sudo journalctl -u k3s

Results:

$ k scale deploy observer --replicas=5 -n test
$ kgp -n test // truncated outputs

pod-97                              1/1     Running     0          10m
test          pod/nginx-pod-98                              1/1     Running     0          10m
test          pod/nginx-pod-99                              1/1     Running     0          10m
test          pod/observer-599c9cb75d-742zv                 1/1     Running     0          9m6s
test          pod/observer-599c9cb75d-m5mgq                 1/1     Running     0          10m
test          pod/observer-599c9cb75d-mg2j9                 1/1     Running     0          9m5s
test          pod/observer-599c9cb75d-ntwb6                 1/1     Running     0          10m
test          pod/observer-599c9cb75d-qn52c                 1/1     Running     0          10m

// logs are looking good for cleaning up the pods without any stuck terminating

time="2023-11-28T18:35:43.027785690Z" level=info msg="RemovePodSandbox \"36066809b7c393f6ddd960e8bfa3d117cae781fc5a6a71aaa01f964a9f0f2cae\" returns successfully"
time="2023-11-28T18:35:43.028525382Z" level=info msg="StopPodSandbox for \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\""
time="2023-11-28T18:35:43.043289573Z" level=info msg="TearDown network for sandbox \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\" successfully"
time="2023-11-28T18:35:43.043318240Z" level=info msg="StopPodSandbox for \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\" returns successfully"
time="2023-11-28T18:35:43.043836067Z" level=info msg="RemovePodSandbox for \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\""
time="2023-11-28T18:35:43.043873471Z" level=info msg="Forcibly stopping sandbox \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\""
time="2023-11-28T18:35:43.069181468Z" level=info msg="TearDown network for sandbox \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\" successfully"
time="2023-11-28T18:35:43.075982888Z" level=warning msg="Failed to get podSandbox status for container event for sandboxID \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a>
time="2023-11-28T18:35:43.076074326Z" level=info msg="RemovePodSandbox \"6947072d68a9b1f7fd1d15d859c7dde9af4f6e61bc6ea6109ea4a6fa0f286f22\" returns successfully"
time="2023-11-28T18:35:43.076630523Z" level=info msg="StopPodSandbox for \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\""
time="2023-11-28T18:35:43.109669919Z" level=info msg="TearDown network for sandbox \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\" successfully"
time="2023-11-28T18:35:43.109700412Z" level=info msg="StopPodSandbox for \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\" returns successfully"
time="2023-11-28T18:35:43.110323728Z" level=info msg="RemovePodSandbox for \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\""
time="2023-11-28T18:35:43.110601791Z" level=info msg="Forcibly stopping sandbox \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\""
time="2023-11-28T18:35:43.130808906Z" level=info msg="TearDown network for sandbox \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\" successfully"
time="2023-11-28T18:35:43.137475711Z" level=warning msg="Failed to get podSandbox status for container event for sandboxID \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75>
time="2023-11-28T18:35:43.137574826Z" level=info msg="RemovePodSandbox \"6e1675d830bce63b7a1e6fe7f90beb43f856566c5ca7b02535f75b70f0a7acab\" returns successfully"
time="2023-11-28T18:35:43.138102730Z" level=info msg="StopPodSandbox for \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\""
time="2023-11-28T18:35:43.150419437Z" level=info msg="TearDown network for sandbox \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\" successfully"
time="2023-11-28T18:35:43.150452619Z" level=info msg="StopPodSandbox for \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\" returns successfully"
time="2023-11-28T18:35:43.151138803Z" level=info msg="RemovePodSandbox for \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\""
time="2023-11-28T18:35:43.151166568Z" level=info msg="Forcibly stopping sandbox \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\""
time="2023-11-28T18:35:43.164044113Z" level=info msg="TearDown network for sandbox \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\" successfully"
time="2023-11-28T18:35:43.170695553Z" level=warning msg="Failed to get podSandbox status for container event for sandboxID \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c225>
time="2023-11-28T18:35:43.170801818Z" level=info msg="RemovePodSandbox \"59d13a9101fb44eab21f500e3f2b0917ff66d40a8cb955d12c2250fd250d7338\" returns successfully"
I1128 18:35:44.311317    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=09a2d11b-6b19-46dd-a39a-57ec06217927 path="/var/lib/kubelet/pods/09a2d11b-6b19->
I1128 18:35:44.312314    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=5a44ffaa-5f1e-4603-b188-2ad26bf7d1ad path="/var/lib/kubelet/pods/5a44ffaa-5f1e->
I1128 18:35:44.313139    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=6da168cb-5dd5-47ca-a2a6-20d5494b7bde path="/var/lib/kubelet/pods/6da168cb-5dd5->
I1128 18:35:44.313583    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=b8a381b5-97fc-4019-a86a-23269c9b0c79 path="/var/lib/kubelet/pods/b8a381b5-97fc->
time="2023-11-28T18:35:45Z" level=debug msg="Wrote ping"
time="2023-11-28T18:35:45Z" level=debug msg="Wrote ping"
time="2023-11-28T18:35:45Z" level=debug msg="Wrote ping"
I1128 18:35:46.311460    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=614da1ca-58bc-4371-8864-58fa6c6c7516 path="/var/lib/kubelet/pods/614da1ca-58bc->
I1128 18:35:46.312124    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=696c2d04-f848-42f9-9257-f2b9e7914e3e path="/var/lib/kubelet/pods/696c2d04-f848->
I1128 18:35:46.312686    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=c8d75035-c3a4-44cd-9ed3-6c0e038b44d7 path="/var/lib/kubelet/pods/c8d75035-c3a4->
I1128 18:35:48.311496    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=a6d64c40-7bde-4165-a2f3-bfc7b14bd3bb path="/var/lib/kubelet/pods/a6d64c40-7bde->
I1128 18:35:48.311932    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=cb02c90c-7aca-4f89-923c-a43a236e6e4f path="/var/lib/kubelet/pods/cb02c90c-7aca->
I1128 18:35:48.312416    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=d5fdeee5-118c-4326-9e06-d9d3e06b46f9 path="/var/lib/kubelet/pods/d5fdeee5-118c->
I1128 18:35:50.309978    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=3cb6b9c3-e6d9-48ed-b275-8b214a41e5e5 path="/var/lib/kubelet/pods/3cb6b9c3-e6d9->
I1128 18:35:50.310612    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=78d628a6-5cb5-4a1b-912d-8f0137a0b7b0 path="/var/lib/kubelet/pods/78d628a6-5cb5->
I1128 18:35:50.311052    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=7a134c04-58ba-4229-9a8f-abcc173ae8a9 path="/var/lib/kubelet/pods/7a134c04-58ba->
I1128 18:35:50.311484    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=e443035d-5064-4cb3-8475-386e397c835a path="/var/lib/kubelet/pods/e443035d-5064->
time="2023-11-28T18:35:50Z" level=debug msg="Wrote ping"
time="2023-11-28T18:35:50Z" level=debug msg="Wrote ping"
time="2023-11-28T18:35:50Z" level=debug msg="Wrote ping"
I1128 18:35:52.311081    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=8a9e263e-407f-4054-92f9-873edd1bded7 path="/var/lib/kubelet/pods/8a9e263e-407f->
I1128 18:35:52.311737    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=adb545b3-8962-4ee6-a846-458ec4f2cb11 path="/var/lib/kubelet/pods/adb545b3-8962->
I1128 18:35:52.312837    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=f57c5fb7-3b83-405d-baf4-6685bb40c09d path="/var/lib/kubelet/pods/f57c5fb7-3b83->
I1128 18:35:54.310050    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=1e15b0c6-4d24-46d2-8f5d-4063d0ae0855 path="/var/lib/kubelet/pods/1e15b0c6-4d24->
I1128 18:35:54.310505    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=66d38a66-3b51-4c3a-97fb-487891e02bec path="/var/lib/kubelet/pods/66d38a66-3b51->
I1128 18:35:54.310885    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=bf494f48-269e-4a84-b613-6976abf9478c path="/var/lib/kubelet/pods/bf494f48-269e->
time="2023-11-28T18:35:55Z" level=debug msg="Wrote ping"
time="2023-11-28T18:35:55Z" level=debug msg="Wrote ping"
time="2023-11-28T18:35:55Z" level=debug msg="Wrote ping"
I1128 18:35:56.311332    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=2b8dbada-4cc6-4113-b001-746386126506 path="/var/lib/kubelet/pods/2b8dbada-4cc6->
I1128 18:35:56.311800    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=3cb61866-f8fb-4a3c-acdb-4956c697cb45 path="/var/lib/kubelet/pods/3cb61866-f8fb->
I1128 18:35:56.312235    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=7b59db6f-d1d6-4774-a6ae-9627fb861e3e path="/var/lib/kubelet/pods/7b59db6f-d1d6->
I1128 18:35:56.312606    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=c3af4e14-1666-440c-8076-9a9c65308e8d path="/var/lib/kubelet/pods/c3af4e14-1666->
I1128 18:35:58.311529    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=2cf3bb5d-2e0e-4426-83fe-3387f97e2997 path="/var/lib/kubelet/pods/2cf3bb5d-2e0e->
I1128 18:35:58.311956    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=2d7c9085-b985-4d1a-968c-2ec2c5e3a229 path="/var/lib/kubelet/pods/2d7c9085-b985->
I1128 18:35:58.312339    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=a0081fd0-72c4-442d-916f-91b2e13e8849 path="/var/lib/kubelet/pods/a0081fd0-72c4->
I1128 18:36:00.314639    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=34343715-3a6f-4a46-ae51-607c7ac19413 path="/var/lib/kubelet/pods/34343715-3a6f->
I1128 18:36:00.315088    5485 kubelet_volumes.go:160] "Cleaned up orphaned pod volumes dir" podUID=c5606499-260b-43f9-819c-8642c422f52e path="/var/lib/kubelet/pods/c5606499-260b->
time="2023-11-28T18:36:00Z" level=debug msg="Wrote ping"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

Labels
None yet
Projects
K3s Development
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brandond @VestigeJ