Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Metrics Server as addon (#3560). #3563

Merged
merged 17 commits into from
Nov 23, 2018
Merged

Support Metrics Server as addon (#3560). #3563

merged 17 commits into from
Nov 23, 2018

Conversation

okamototk
Copy link
Contributor

Hi,

I send pull request about metrics server. But I wonder whether you prefer include it or not.
IMO,　autoscale doesn't work with metrics server and it worth to include main playbook and it worth to include main.

Or should I include it in contrib? I would like to hear your opinion.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 21, 2018
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 21, 2018
@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

7 similar comments
@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@woopstar
Copy link
Member

Does this PR work, if you apply #3465 ?

@okamototk
Copy link
Contributor Author

Does this PR work, if you apply #3465 ?

No, I got following error after apply #3465.

$  kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .
Error from server (ServiceUnavailable): the server is currently unable to handle the request

@okamototk
Copy link
Contributor Author

okamototk commented Oct 22, 2018 •
edited
Loading

Does this PR work, if you apply #3465 ?

No, I got following error. Should I fix it?

$ kubectl  logs    pod/metrics-server-56c54bf5fc-rmxwm -nkube-system
I1021 10:21:46.311598       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W1021 10:21:57.574621       1 authentication.go:245] Unable to get configmap/extension-apiserver-authentication in kube-system.  Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
Error: Get https://10.233.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.233.0.1:443: connect: connection refused
Usage:
   [flags]

Flags:
      --alsologtostderr                                         log to standard error as well as files
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create  subjectaccessreviews.authorization.k8s.io.
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
      --enable-swagger-ui                                       Enables swagger ui on the apiserver at /swagger-ui
  -h, --help                                                    help for this command
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
      --kubeconfig string                                       The path to the kubeconfig used to connect to the Kubernetes API server and the Kubelets (defaults to in-cluster config)
      --kubelet-insecure-tls                                    Do not verify CA of serving certificates presented by Kubelets.  For testing purposes only.
      --kubelet-port int                                        The port to use to connect to Kubelets (defaults to 10250) (default 10250)
      --kubelet-preferred-address-types strings                 The priority of node address types to use when determining which address to use to connect to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])
      --log-flush-frequency duration                            Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                          when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                          If non-empty, write log files in this directory
      --logtostderr                                             log to standard error instead of files (default true)
      --metric-resolution duration                              The resolution at which metrics-server will retain metrics. (default 1m0s)
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --requestheader-allowed-names strings                     List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
      --requestheader-extra-headers-prefix strings              List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers strings                     List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers strings                  List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
      --stderrthreshold severity                                logs at or above this threshold go to stderr (default 2)
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      --tls-cipher-suites strings                               Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use.  Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA
      --tls-min-version string                                  Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
  -v, --v Level                                                 log level for V logs
      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging

panic: Get https://10.233.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.233.0.1:443: connect: connection refused

goroutine 1 [running]:
main.main()
        /go/src/github.com/kubernetes-incubator/metrics-server/cmd/metrics-server/metrics-server.go:39 +0x13b

@woopstar
Copy link
Member

You need to make that work , as it will become defaults. It is prolly due to removing the insecure port.

@okamototk
Copy link
Contributor Author

You need to make that work , as it will become defaults. It is prolly due to removing the insecure port.

Sure, but I can't find the cause, If you know, could you help me ?

Messages said:

W1021 10:21:57.574621 1 authentication.go:245] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'

So I think resolution is adding extension-apiserver-authentication-reader role.

But when I checked rolebindings, it had it as following:

$ kubectl get  rolebindings/metrics-server-auth-reader  -nkube-system -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"RoleBinding","metadata":{"annotations":{},"name":"metrics-server-auth-reader","namespace":"kube-system"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"extension-apiserver-authentication-reader"},"subjects":[{"kind":"ServiceAccount","name":"metrics-server","namespace":"kube-system"}]}
  creationTimestamp: 2018-10-21T09:52:24Z
  name: metrics-server-auth-reader
  namespace: kube-system
  resourceVersion: "1561"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/metrics-server-auth-reader
  uid: 0326d2e7-d517-11e8-9b67-000c299cb260
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

So config had extension-apiserver-authentication-reader already.

  kind: Role
  name: extension-apiserver-authentication-reader

Do you have any idea?

@woopstar
Copy link
Member

You need to make that work , as it will become defaults. It is prolly due to removing the insecure port.

Sure, but I can't find the cause, If you know, could you help me ?

Messages said:

W1021 10:21:57.574621 1 authentication.go:245] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLE_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'

So I think resolution is adding extension-apiserver-authentication-reader role.

But when I checked rolebindings, it had it as following:

$ kubectl get  rolebindings/metrics-server-auth-reader  -nkube-system -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"RoleBinding","metadata":{"annotations":{},"name":"metrics-server-auth-reader","namespace":"kube-system"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"extension-apiserver-authentication-reader"},"subjects":[{"kind":"ServiceAccount","name":"metrics-server","namespace":"kube-system"}]}
  creationTimestamp: 2018-10-21T09:52:24Z
  name: metrics-server-auth-reader
  namespace: kube-system
  resourceVersion: "1561"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/metrics-server-auth-reader
  uid: 0326d2e7-d517-11e8-9b67-000c299cb260
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

So config had extension-apiserver-authentication-reader already.

  kind: Role
  name: extension-apiserver-authentication-reader

Do you have any idea?

I might be due to the fact that the read-only port is disabled. Our metrics-server deployment is as following:

containers:
      - name: metrics-server
        image: gcr.io/google_containers/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true

It requires:

kubelet_authentication_token_webhook: true
kubelet_authorization_mode_webhook: false

Which should be default though

@okamototk
Copy link
Contributor Author

@woopstar

Thanks, I updated. I can't find source option for metrcis-server v0.3.1. Currently, I use previous v0.2.1.

@okamototk okamototk mentioned this pull request Oct 24, 2018
Closed
@okamototk
Copy link
Contributor Author

Now it's based on metrics server v0.3.1 and it shoud work with PR #3465.

woopstar
woopstar suggested changes Oct 26, 2018
View reviewed changes
Copy link
Member

@woopstar woopstar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please enable in CI test addons too:

https://github.com/kubernetes-incubator/kubespray/blob/master/tests/files/gce_centos7-flannel-addons.yml

roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2 Show resolved Hide resolved
roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2 Outdated Show resolved Hide resolved
roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2 Outdated Show resolved Hide resolved
roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2 Outdated Show resolved Hide resolved
roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2 Outdated Show resolved Hide resolved
roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2 Outdated Show resolved Hide resolved
roles/kubernetes-apps/metrics_server/templates/metrics-server-service.yaml.j2 Outdated Show resolved Hide resolved
@woopstar
Copy link
Member

ci check this

@okamototk
Copy link
Contributor Author

@woopstar

Sorry, following multiple entries for kubelet_preferred_address_types does not work default configuration.

Can this be multiple entries like the kubelet_preferred_address_types: 'InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP' ? Because then I would suggest to set it equally

I would like to specify just InternalIP as default.

@okamototk
Make InternalIP default for metrics_server_kubelet_preferred_address_…
901f539 
…types

Make InternalIP default because multiple preferrred address types does not work.
@woopstar
Copy link
Member

woopstar commented Nov 1, 2018

Testing on a CoreOS node here seems not to work:

/opt/bin/kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   coredns-79d7c86b44-4cxgq                1/1     Running   0          4m27s
kube-system   coredns-79d7c86b44-tr4sw                0/1     Pending   0          4m27s
kube-system   kube-apiserver-node1                    1/1     Running   0          6m38s
kube-system   kube-controller-manager-node1           1/1     Running   0          6m38s
kube-system   kube-proxy-wctjq                        1/1     Running   0          4m59s
kube-system   kube-scheduler-node1                    1/1     Running   0          6m38s
kube-system   kubernetes-dashboard-5db4d9f45f-fdvjp   1/1     Running   0          4m19s
kube-system   metrics-server-7f9c7c6584-v869m         0/2     Pending   0          29s
kube-system   tiller-deploy-564ddd748f-z44n7          0/1     Pending   0          4m1s
kube-system   weave-net-28hgz                         2/2     Running   0          5m3s

Here is the settings used . I'm running from master branch:

diff --git a/inventory/sample/group_vars/all/all.yml b/inventory/sample/group_vars/all/all.yml
index dbe608fa..580b8430 100644
--- a/inventory/sample/group_vars/all/all.yml
+++ b/inventory/sample/group_vars/all/all.yml
@@ -33,9 +33,9 @@ bin_dir: /usr/local/bin
 #kubelet_load_modules: false

 ## Upstream dns servers used by dnsmasq
-#upstream_dns_servers:
-#  - 8.8.8.8
-#  - 8.8.4.4
+upstream_dns_servers:
+  - 8.8.8.8
+  - 8.8.4.4

 ## There are some changes specific to the cloud providers
 ## for instance we need to encapsulate packets with some network plugins
@@ -46,7 +46,7 @@ bin_dir: /usr/local/bin


 ## Uncomment to enable experimental kubeadm deployment mode
-#kubeadm_enabled: false
+kubeadm_enabled: true

 ## Set these proxy values in order to update package manager and docker daemon to use proxies
 #http_proxy: ""
diff --git a/inventory/sample/group_vars/k8s-cluster/addons.yml b/inventory/sample/group_vars/k8s-cluster/addons.yml
index ca801d3c..e85b900f 100644
--- a/inventory/sample/group_vars/k8s-cluster/addons.yml
+++ b/inventory/sample/group_vars/k8s-cluster/addons.yml
@@ -3,7 +3,7 @@
 dashboard_enabled: true

 # Helm deployment
-helm_enabled: false
+helm_enabled: true

 # Registry deployment
 registry_enabled: false
@@ -12,7 +12,7 @@ registry_enabled: false
 # registry_disk_size: "10Gi"

 # Metrics Server deployment
-metrics_server_enabled: false
+metrics_server_enabled: true
 # metrics_server_kubelet_insecure_tls: true
 # metrics_server_metric_resolution: 60s
 # metrics_server_kubelet_preferred_address_types: "InternalIP"
diff --git a/inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml b/inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml
index 0279c7c1..ad87b2fb 100644
--- a/inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml
+++ b/inventory/sample/group_vars/k8s-cluster/k8s-cluster.yml
@@ -71,7 +71,7 @@ kube_users:

 # Choose network plugin (cilium, calico, contiv, weave or flannel)
 # Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
-kube_network_plugin: calico
+kube_network_plugin: weave

 # Kubernetes internal network for services, unused block of space.
 kube_service_addresses: 10.233.0.0/18
@@ -89,13 +89,13 @@ kube_network_node_prefix: 24
 # The port the API Server will be listening on.
 kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
 kube_apiserver_port: 6443 # (https)
-kube_apiserver_insecure_port: 8080 # (http)
+#kube_apiserver_insecure_port: 8080 # (http)
 # Set to 0 to disable insecure port - Requires RBAC in authorization_modes and kube_api_anonymous_auth: true
-#kube_apiserver_insecure_port: 0 # (disabled)
+kube_apiserver_insecure_port: 0 # (disabled)

 # Kube-proxy proxyMode configuration.
 # Can be ipvs, iptables
-kube_proxy_mode: iptables
+kube_proxy_mode: ipvs

 # Kube-proxy nodeport address.
 # cidr to bind nodeport services. Flag --nodeport-addresses on kube-proxy manifest
@@ -103,7 +103,7 @@ kube_proxy_nodeport_addresses: false
 # kube_proxy_nodeport_addresses_cidr: 10.0.1.0/24

 ## Encrypting Secret Data at Rest (experimental)
-kube_encrypt_secret_data: false
+kube_encrypt_secret_data: true

 # DNS configuration.
 # Kubernetes cluster name, also will be used as DNS domain
@@ -111,12 +111,12 @@ cluster_name: cluster.local
 # Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
 ndots: 2
 # Can be dnsmasq_kubedns, kubedns, coredns, coredns_dual, manual or none
-dns_mode: kubedns
+dns_mode: coredns
 # Set manual server if using a custom cluster DNS server
 #manual_dns_server: 10.x.x.x

 # Can be docker_dns, host_resolvconf or none
-resolvconf_mode: docker_dns
+resolvconf_mode: none
 # Deploy netchecker app to verify DNS resolve as an HTTP service
 deploy_netchecker: false
 # Ip address of the kubernetes skydns service
@@ -139,7 +139,7 @@ helm_deployment_type: host
 k8s_image_pull_policy: IfNotPresent

 # audit log for kubernetes
-kubernetes_audit: false
+kubernetes_audit: true

 # dynamic kubelet configuration
 dynamic_kubelet_configuration: false
@@ -167,7 +167,7 @@ podsecuritypolicy_enabled: false

 # A comma separated list of levels of node allocatable enforcement to be enforced by kubelet.
 # Acceptable options are 'pods', 'system-reserved', 'kube-reserved' and ''. Default is "".
-# kubelet_enforce_node_allocatable: pods
+kubelet_enforce_node_allocatable: pods

 ## Supplementary addresses that can be added in kubernetes ssl keys.
 ## That can be useful for example to setup a keepalived virtual IP
diff --git a/inventory/sample/hosts.ini b/inventory/sample/hosts.ini
index 8e32a3a7..055483f9 100644
--- a/inventory/sample/hosts.ini
+++ b/inventory/sample/hosts.ini
@@ -1,32 +1,14 @@
-# ## Configure 'ip' variable to bind kubernetes services on a
-# ## different ip than the default iface
-# ## We should set etcd_member_name for etcd cluster. The node that is not a etcd member do not need to set the value, or can set the empty string value.
 [all]
-# node1 ansible_host=95.54.0.12  # ip=10.3.0.1 etcd_member_name=etcd1
-# node2 ansible_host=95.54.0.13  # ip=10.3.0.2 etcd_member_name=etcd2
-# node3 ansible_host=95.54.0.14  # ip=10.3.0.3 etcd_member_name=etcd3
-# node4 ansible_host=95.54.0.15  # ip=10.3.0.4 etcd_member_name=etcd4
-# node5 ansible_host=95.54.0.16  # ip=10.3.0.5 etcd_member_name=etcd5
-# node6 ansible_host=95.54.0.17  # ip=10.3.0.6 etcd_member_name=etcd6
-
-# ## configure a bastion host if your nodes are not directly reachable
-# bastion ansible_host=x.x.x.x ansible_user=some_user
+node1 ansible_host=10.50.61.199 ansible_ssh_port=34 ansible_ssh_user=core

 [kube-master]
-# node1
-# node2
+node1

 [etcd]
-# node1
-# node2
-# node3
+node1

 [kube-node]
-# node2
-# node3
-# node4
-# node5
-# node6
+node1

 [k8s-cluster:children]
 kube-master

And no logs:

opt/bin/kubectl logs -n kube-system metrics-server-7f9c7c6584-v869m metrics-server-nanny
node1 ak #

/opt/bin/kubectl logs -n kube-system metrics-server-7f9c7c6584-v869m metrics-server
node1 ak #

Here is the describes:

/opt/bin/kubectl describe -n kube-system deploy/metrics-server
Name:                   metrics-server
Namespace:              kube-system
CreationTimestamp:      Thu, 01 Nov 2018 17:19:45 +0100
Labels:                 addonmanager.kubernetes.io/mode=Reconcile
                        app.kubernetes.io/name=metrics-server
                        kubernetes.io/cluster-service=true
                        version=v0.3.1
Annotations:            deployment.kubernetes.io/revision: 2
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile...
Selector:               app.kubernetes.io/name=metrics-server,version=v0.3.1
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:           app.kubernetes.io/name=metrics-server
                    version=v0.3.1
  Annotations:      scheduler.alpha.kubernetes.io/critical-pod:
                    seccomp.security.alpha.kubernetes.io/pod: docker/default
  Service Account:  metrics-server
  Containers:
   metrics-server:
    Image:      k8s.gcr.io/metrics-server-amd64:v0.3.1
    Port:       443/TCP
    Host Port:  0/TCP
    Command:
      /metrics-server
      --kubelet-preferred-address-types=InternalIP
      --kubelet-insecure-tls
      --metric-resolution=60s
    Liveness:     http-get https://:https/healthz delay=30s timeout=10s period=30s #success=1 #failure=3
    Readiness:    http-get https://:443/healthz delay=30s timeout=10s period=30s #success=1 #failure=3
    Environment:  <none>
    Mounts:       <none>
   metrics-server-nanny:
    Image:      k8s.gcr.io/addon-resizer:1.8.3
    Port:       <none>
    Host Port:  <none>
    Command:
      /pod_nanny
      --config-dir=/etc/config
      --cpu=40m
      --extra-cpu=0.5m
      --memory=35Mi
      --extra-memory=4Mi
      --threshold=5
      --deployment=metrics-server-v0.3.1
      --container=metrics-server
      --poll-period=300000
      --estimator=exponential
      --minClusterSize=5
    Limits:
      cpu:     100m
      memory:  300Mi
    Requests:
      cpu:     5m
      memory:  50Mi
    Environment:
      MY_POD_NAME:        (v1:metadata.name)
      MY_POD_NAMESPACE:   (v1:metadata.namespace)
    Mounts:
      /etc/config from metrics-server-config-volume (rw)
  Volumes:
   metrics-server-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      metrics-server-config
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
OldReplicaSets:  <none>
NewReplicaSet:   metrics-server-7f9c7c6584 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  7m13s  deployment-controller  Scaled up replica set metrics-server-64c9f4d79b to 1
  Normal  ScalingReplicaSet  3m59s  deployment-controller  Scaled up replica set metrics-server-7f9c7c6584 to 1
  Normal  ScalingReplicaSet  3m59s  deployment-controller  Scaled down replica set metrics-server-64c9f4d79b to 0

/opt/bin/kubectl describe -n kube-system pod/metrics-server-7f9c7c6584-v869m
Name:               metrics-server-7f9c7c6584-v869m
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               <none>
Labels:             app.kubernetes.io/name=metrics-server
                    pod-template-hash=7f9c7c6584
                    version=v0.3.1
Annotations:        scheduler.alpha.kubernetes.io/critical-pod:
                    seccomp.security.alpha.kubernetes.io/pod: docker/default
Status:             Pending
IP:
Controlled By:      ReplicaSet/metrics-server-7f9c7c6584
Containers:
  metrics-server:
    Image:      k8s.gcr.io/metrics-server-amd64:v0.3.1
    Port:       443/TCP
    Host Port:  0/TCP
    Command:
      /metrics-server
      --kubelet-preferred-address-types=InternalIP
      --kubelet-insecure-tls
      --metric-resolution=60s
    Liveness:     http-get https://:https/healthz delay=30s timeout=10s period=30s #success=1 #failure=3
    Readiness:    http-get https://:443/healthz delay=30s timeout=10s period=30s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-shm7q (ro)
  metrics-server-nanny:
    Image:      k8s.gcr.io/addon-resizer:1.8.3
    Port:       <none>
    Host Port:  <none>
    Command:
      /pod_nanny
      --config-dir=/etc/config
      --cpu=40m
      --extra-cpu=0.5m
      --memory=35Mi
      --extra-memory=4Mi
      --threshold=5
      --deployment=metrics-server-v0.3.1
      --container=metrics-server
      --poll-period=300000
      --estimator=exponential
      --minClusterSize=5
    Limits:
      cpu:     100m
      memory:  300Mi
    Requests:
      cpu:     5m
      memory:  50Mi
    Environment:
      MY_POD_NAME:       metrics-server-7f9c7c6584-v869m (v1:metadata.name)
      MY_POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /etc/config from metrics-server-config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-shm7q (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  metrics-server-config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      metrics-server-config
    Optional:  false
  metrics-server-token-shm7q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-shm7q
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  54s (x25 over 4m35s)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.

The issue is, this toleration is not added:
https://github.com/kubernetes-incubator/kubespray/pull/3563/files#diff-8336e707cb049d7c102f78ec9db47097R120

Because the node has the following tolerations:

Taints:             node-role.kubernetes.io/master:NoSchedule

This is because kubeadm addes the taint. Apparently this task is not run correctly upon provision: https://github.com/kubernetes-incubator/kubespray/blob/master/roles/kubernetes/master/tasks/kubeadm-setup.yml#L214

@mattymo Is this due to failed_when: false ?

Because I do have this in my play:

TASK [kubernetes/master : kubeadm | Remove taint for master with node role] ****************************************************************************************************************************
Thursday 01 November 2018  16:16:41 +0000 (0:00:00.072)       0:07:37.111 *****
changed: [node1 -> 10.50.61.199]

@okamototk okamototk mentioned this pull request Nov 2, 2018
Closed
@okamototk
Copy link
Contributor Author

okamototk commented Nov 2, 2018 •
edited
Loading

This is due to kube config is required when remove insecure port.
This issue is fixed following PR.

#3461

@okamototk
Copy link
Contributor Author

okamototk commented Nov 21, 2018 •
edited
Loading

Depended PR #3461 was merged. Now this PR cloud be merged.

@woopstar
Copy link
Member

Please confirm it still works if #3465 is merged. Especially when kube_apiserver_insecure_port is disabled

@okamototk
Copy link
Contributor Author

Please confirm it still works if #3465 is merged. Especially when kube_apiserver_insecure_port is disabled

I confirmed. It worked.

@woopstar
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 23, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: woopstar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 23, 2018
@k8s-ci-robot k8s-ci-robot merged commit c5e425b into kubernetes-sigs:master Nov 23, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@woopstar woopstar woopstar approved these changes

@chapsuk chapsuk Awaiting requested review from chapsuk

@jjungnickel jjungnickel Awaiting requested review from jjungnickel

Assignees

@woopstar woopstar

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@okamototk @woopstar @k8s-ci-robot