kubelet.service: Wait for network-online.target

[rosti]

What type of PR is this?

/kind bug

What this PR does / why we need it:

Whenever kubeadm detects a system that has systemd-resolved running, it would
provision the kubelet on the local node with a resolv.conf overwrite -
/run/systemd/resolve/resolv.conf.

However, some kubeadm users have discovered an issue during system boot.
The kubelet can end up in a race with the systemd-resolved service and actually
startup loads with empty or incorrect resolve.conf files.

The race is caused by the fact that the kubelet.service file does not indicate
dependence on the network-online.target.

To fix this we add network-online.target as a dependency and wait for its
initialization to complete before starting the kubelet.

Which issue(s) this PR fixes:

Refs #1248, kubernetes/kubeadm#2111

Special notes for your reviewer:

/cc @neolit123
/cc @kubernetes/release-engineering
/cc @kubernetes/sig-node-bugs
/assign @justaugustus @tpepper
/priority important-longterm

Does this PR introduce a user-facing change?

systemd would now start the kubelet service after the network-online.target is reached.

[neolit123]

/lgtm
cc @chrisohaver @rajansandeep @BenTheElder @aojea @hickeng

[ncdc]

cc @detiber

[saschagrunert]

/lgtm

But I'm wondering if this fixes the real issue since it seemed related to kubeadm.

[rosti]

But I'm wondering if this fixes the real issue since it seemed related to kubeadm.

@saschagrunert there are a couple of problems in the original kubeadm issue:

1. kubeadm ignores resolv.conf overwrite by users if they are done through the kubelet component config (and not the kubelet command line). That problem requires a patch in kubeadm itself.

2. Users with slow systems, who rely on kubeadm to point the kubelet to the systemd-resolved resolv.conf file, might experience problems due to the described race (the problem this PR fixes).

[dims]

cc @akutz

[SataQiu]

/lgtm
👍

[hickeng]

@rosti I think this is a useful change and does close at least one timing hole.
However please note that it doesn't address kubernetes/kubeadm#2111 which comes from a post-boot restart of systemd-networkd.

[justaugustus]

Thanks for the fix!
/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, neolit123, rosti, saschagrunert, SataQiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kubepkg/OWNERS [justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment