[BUG] yurtadm join failed:dial tcp 127.0.0.1:10261: connect: connection refused

[win32fk]

kubenetes version : v1.28.0
openyurt versin : yurtadm-v1.4.4

What happened:
在边缘机器，执行openYurt join失败 ：xx.xx.xx.xx is my ip
execute openyurt join xx.xx.xx.xx:6443 --token=4etrp2.p101y8dgjm98b9b0 --node-type=edge --discovery-token-unsafe-skip-ca-verification --v=5 --cri-socket=/run/containerd/containerd.sock
but its failed:
execution phase kubelet-start: error uploading crisocket: Get "http://127.0.0.1:10261/api/v1/nodes/gm-system-product-name?timeout=10s": dial tcp 127.0.0.1:10261: connect: connection refused

logs:
I0704 14:29:08.852401 39105 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "xx.xx.xx.xx:6443"
I0704 14:29:08.852421 39105 token.go:230] [discovery] kube client is nil, use bootstrap config(&{ {false map[]} map[kubernetes:0xc0007a1700] map[] map[token-bootstrap-client@kubernetes:0xc00079e5a0] token-bootstrap-client@kubernetes map[]}) to create client
I0704 14:29:08.876802 39105 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "xx.xx.xx.xx:6443"
I0704 14:29:08.898081 39105 kubernetes.go:369] kubernetes version: v1.28.0
I0704 14:29:08.898140 39105 join.go:396] node join data info: join.joinData{cfgPath:"", joinNodeData:(*joindata.NodeRegistration)(0xc00079e550), apiServerEndpoint:"xx.xx.xx.xx6443", token:"4etrp2.p101y8dgjm98b9b0", tlsBootstrapCfg:(*api.Config)(0xc00057a5a0), clientSet:(*kubernetes.Clientset)(0xc0006002c0), ignorePreflightErrors:sets.String{"DirAvailable--etc-kubernetes-manifests":sets.Empty{}, "FileAvailable--etc-kubernetes-kubelet.conf":sets.Empty{}}, organizations:"", pauseImage:"registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2", yurthubImage:"registry.cn-hangzhou.aliyuncs.com/openyurt/yurthub:latest", yurthubTemplate:"", yurthubManifest:"", kubernetesVersion:"v1.28.0", caCertHashes:[]string{}, nodeLabels:map[string]string{}, kubernetesResourceServer:"dl.k8s.io", yurthubServer:"127.0.0.1", reuseCNIBin:false, namespace:"kube-system", staticPodTemplateList:[]string(nil), staticPodManifestList:[]string(nil)}
I0704 14:29:08.904489 39105 system.go:43] Setting ipv4 forward
I0704 14:29:08.904567 39105 system.go:52] Setting bridge settings for kubernetes.
I0704 14:29:08.904709 39105 system.go:74] Disabling SELinux.
I0704 14:29:08.904721 39105 kubernetes.go:117] Check and install kubelet v1.28.0
I0704 14:29:08.942215 39105 kubernetes.go:126] kubelet --version: v1.28.0
I0704 14:29:08.942243 39105 kubernetes.go:138] Kubelet v1.28.0 already exist, skip install.
I0704 14:29:08.942246 39105 kubernetes.go:208] Check and install kubeadm v1.28.0
I0704 14:29:08.961953 39105 kubernetes.go:215] "kubeadm" version="{\n "clientVersion": {\n "major": "1",\n "minor": "28",\n "gitVersion": "v1.28.0",\n "gitCommit": "855e7c48de7388eb330da0f8d9d2394ee818fb8d",\n "gitTreeState": "clean",\n "buildDate": "2023-08-15T10:20:15Z",\n "goVersion": "go1.20.7",\n "compiler": "gc",\n "platform": "linux/amd64"\n }\n}\n"
I0704 14:29:08.962009 39105 kubernetes.go:232] Kubeadm v1.28.0 already exist, skip install.
I0704 14:29:08.962021 39105 kubernetes.go:183] Skip download cni, use already exist file: /tmp/cni-plugins-linux-amd64-v0.8.0.tgz
I0704 14:29:09.237897 39105 kubernetes.go:258] Setting kubelet service.
I0704 14:29:09.241044 39105 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "xx.xx.xx.xx:6443"
I0704 14:29:09.241050 39105 token.go:230] [discovery] kube client is nil, use bootstrap config(&{ {false map[]} map[kubernetes:0xc000022080] map[] map[token-bootstrap-client@kubernetes:0xc00079e9b0] token-bootstrap-client@kubernetes map[]}) to create client
I0704 14:29:09.247318 39105 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "xx.xx.xx.xx:6443"
I0704 14:29:09.247611 39105 yurthub.go:44] [join-node] Adding edge hub static yaml
I0704 14:29:09.247652 39105 yurthub.go:88] yurthub template: /etc/kubernetes/manifests/yurthub.yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
openyurt.io/static-pod-hash: 6c64fbdf49
creationTimestamp: null
labels:
k8s-app: yurt-hub
name: yurt-hub
namespace: kube-system
spec:
containers:

• command:
  • yurthub
  • --v=2
  • --bind-address=127.0.0.1
  • --server-addr=https://xx.xx.xx.xx:6443
  • --node-name=$(NODE_NAME)
  • --bootstrap-file=/var/lib/yurthub/bootstrap-hub.conf
  • --working-mode=edge
  • --namespace=kube-system
    env:
  • name: NODE_NAME
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: spec.nodeName
    image: openyurt/yurthub:v1.4.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
    failureThreshold: 3
    httpGet:
    host: 127.0.0.1
    path: /v1/healthz
    port: 10267
    scheme: HTTP
    initialDelaySeconds: 300
    periodSeconds: 5
    successThreshold: 1
    timeoutSeconds: 1
    name: yurt-hub
    resources:
    limits:
    memory: 300Mi
    requests:
    cpu: 150m
    memory: 150Mi
    securityContext:
    capabilities:
    add:
    • NET_ADMIN
    • NET_RAW
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
  • mountPath: /var/lib/yurthub
    name: hub-dir
  • mountPath: /etc/kubernetes
    name: kubernetes
    dnsPolicy: ClusterFirst
    hostNetwork: true
    priority: 2000001000
    priorityClassName: system-node-critical
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    terminationGracePeriodSeconds: 30
    volumes:
• hostPath:
  path: /var/lib/yurthub
  type: DirectoryOrCreate
  name: hub-dir
• hostPath:
  path: /etc/kubernetes
  type: Directory
  name: kubernetes
  status: {}
  I0704 14:29:09.247681 39105 yurthub.go:93] [join-node] Add hub agent static yaml is ok
  I0704 14:29:09.247686 39105 token.go:105] [discovery] Created cluster-info discovery client, requesting info from "xx.xx.xx.xx:6443"
  I0704 14:29:09.247689 39105 token.go:230] [discovery] kube client is nil, use bootstrap config(&{ {false map[]} map[kubernetes:0xc0005ffe00] map[] map[token-bootstrap-client@kubernetes:0xc0000b6500] token-bootstrap-client@kubernetes map[]}) to create client
  I0704 14:29:09.253500 39105 token.go:130] [discovery] Cluster info signature and contents are valid and no TLS pinning was specified, will use API Server "xx.xx.xx.xx:6443"
  W0704 14:29:09.300763 39154 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
  [preflight] Running pre-flight checks
  [WARNING FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
  [preflight] Reading configuration from the cluster...
  [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
  [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  [kubelet-start] Starting the kubelet
  [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
  [kubelet-check] Initial timeout of 40s passed.
  error execution phase kubelet-start: error uploading crisocket: Get "http://127.0.0.1:10261/api/v1/nodes/gm-system-product-name?timeout=10s": dial tcp 127.0.0.1:10261: connect: connection refused
  To see the stack trace of this error execute with --v=5 or higher
  Error: exit status 1

kubelet logs:

7月 04 14:32:22 gm-System-Product-Name kubelet[39330]: E0704 14:32:22.512666 39330 controller.go:146] "Failed to ensure lease exists, will retry" err="Get "http://127.0.0.1:10261/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/gm-system-product-name?timeout=10s\": dial tcp 127.0.0.1:10261: connect: connection refused" interval="7s"
7月 04 14:32:22 gm-System-Product-Name kubelet[39330]: I0704 14:32:22.668020 39330 kubelet_node_status.go:70] "Attempting to register node" node="gm-system-product-name"
7月 04 14:32:22 gm-System-Product-Name kubelet[39330]: E0704 14:32:22.668664 39330 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "http://127.0.0.1:10261/api/v1/nodes\": dial tcp 127.0.0.1:10261: connect: connection refused" node="gm-system-product-name"
7月 04 14:32:24 gm-System-Product-Name kubelet[39330]: W0704 14:32:24.117514 39330 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIDriver: Get "http://127.0.0.1:10261/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 127.0.0.1:10261: connect: connection refused
7月 04 14:32:24 gm-System-Product-Name kubelet[39330]: E0704 14:32:24.117633 39330 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "http://127.0.0.1:10261/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 127.0.0.1:10261: connect: connection refused
7月 04 14:32:24 gm-System-Product-Name kubelet[39330]: E0704 14:32:24.943753 39330 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "gm-system-product-name" not found"
7月 04 14:32:25 gm-System-Product-Name kubelet[39330]: E0704 14:32:25.234966 39330 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"gm-system-product-name.17deeeb1be6e9311", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"gm-system-product-name", UID:"gm-system-product-name", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"InvalidDiskCapacity", Message:"invalid capacity 0 on image filesystem", Source:v1.EventSource{Component:"kubelet", Host:"gm-system-product-name"}, FirstTimestamp:time.Date(2024, time.July, 4, 14, 29, 14, 873647889, time.Local), LastTimestamp:time.Date(2024, time.July, 4, 14, 29, 14, 873647889, time.Local), Count:1, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"kubelet", ReportingInstance:"gm-system-product-name"}': 'Post "http://127.0.0.1:10261/api/v1/namespaces/default/events": dial tcp 127.0.0.1:10261: connect: connection refused'(may retry after sleeping)
7月 04 14:32:29 gm-System-Product-Name kubelet[39330]: E0704 14:32:29.513976 39330 controller.go:146] "Failed to ensure lease exists, will retry" err="Get "http://127.0.0.1:10261/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/gm-system-product-name?timeout=10s\": dial tcp 127.0.0.1:10261: connect: connection refused" interval="7s"
7月 04 14:32:29 gm-System-Product-Name kubelet[39330]: I0704 14:32:29.670373 39330 kubelet_node_status.go:70] "Attempting to register node" node="gm-system-product-name"
7月 04 14:32:29 gm-System-Product-Name kubelet[39330]: E0704 14:32:29.670958 39330 kubelet_node_status.go:92] "Unable to register node with API server" err="Post "http://127.0.0.1:10261/api/v1/nodes\": dial tcp 127.0.0.1:10261: connect: connection refused" node="gm-system-product-name"
7月 04 14:32:34 gm-System-Product-Name kubelet[39330]: E0704 14:32:34.944939 39330 eviction_manager.go:258] "Eviction manager: failed to get summary stats" err="failed to get node info: node "gm-system-product-name" not found"

Environment:

• OpenYurt version:yurtadm-v1.4.4
• Kubernetes version (use kubectl version):v.1.28.0
• OS (e.g: cat /etc/os-release):Ubuntu 24.02

/kind bug

[win32fk]

openyurt/yurthub:v1.4.0镜像没有 拉下来，后边手动下载镜像，走本地，可以了。