kubelet.service 使用证书续订后失败
kubelet.service is getting failed after using certificate renew
我正在尝试更新过期的证书，我按照以下步骤操作，但 kubectl 服务开始失败。我是 kubernetes 的新手，请帮助我。
# kubeadm alpha certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
admin.conf Nov 11, 2020 12:52 UTC <invalid> no
apiserver Nov 11, 2020 12:52 UTC <invalid> no
apiserver-etcd-client Nov 11, 2020 12:52 UTC <invalid> no
apiserver-kubelet-client Nov 11, 2020 12:52 UTC <invalid> no
controller-manager.conf Nov 11, 2020 12:52 UTC <invalid> no
etcd-healthcheck-client Nov 11, 2020 12:52 UTC <invalid> no
etcd-peer Nov 11, 2020 12:52 UTC <invalid> no
etcd-server Nov 11, 2020 12:52 UTC <invalid> no
front-proxy-client Nov 11, 2020 12:52 UTC <invalid> no
scheduler.conf Nov 11, 2020 12:52 UTC <invalid> no
# kubeadm alpha certs renew all
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
# kubeadm alpha certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
admin.conf Nov 17, 2021 05:49 UTC 364d no
apiserver Nov 17, 2021 05:49 UTC 364d no
apiserver-etcd-client Nov 17, 2021 05:49 UTC 364d no
apiserver-kubelet-client Nov 17, 2021 05:49 UTC 364d no
controller-manager.conf Nov 17, 2021 05:49 UTC 364d no
etcd-healthcheck-client Nov 17, 2021 05:49 UTC 364d no
etcd-peer Nov 17, 2021 05:49 UTC 364d no
etcd-server Nov 17, 2021 05:49 UTC 364d no
front-proxy-client Nov 17, 2021 05:49 UTC 364d no
scheduler.conf Nov 17, 2021 05:49 UTC 364d no
:~> mkdir -p $HOME/.kube
:~> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
:~> sudo chown $(id -u):$(id -g) $HOME/.kube/config
:~> sudo systemctl daemon-reload
:~> sudo systemctl stop kubelet
:~> sudo systemctl start kubelet
:~> sudo systemctl enable kubelet
:~> sudo systemctl stop docker
:~> sudo systemctl start docker
:~> kubectl get pods
The connection to the server 10.xx.xx.74:6443 was refused - did you specify the right host or port?
# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean"GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 10.xx.xx.74:6443 was refused - did you specify the right host or port?
Kubectl 状态：
# systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2020-11-17 08:18:20 UTC; 1s ago
Docs: https://kubernetes.io/docs/
Process: 1452 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, sta
Main PID: 1452 (code=exited, status=255)
Nov 17 08:18:20 c536gocrb systemd[1]: Unit kubelet.service entered failed state.
Nov 17 08:18:20 c536gocrb systemd[1]: kubelet.service failed.
尝试将环境变量添加到管理员 10.kubeadm.conf
环境="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"
kubeadm 配置文件：
cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
中的建议替换 kubelet.conf 中的证书数据后
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ==
server: https://xx.x.x.x.:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: system:node:cmaster
name: system:node:cmaster@kubernetes
current-context: system:node:cmaster@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:cmaster
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
我正在尝试更新过期的证书，我按照以下步骤操作，但 kubectl 服务开始失败。我是 kubernetes 的新手，请帮助我。
# kubeadm alpha certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
admin.conf Nov 11, 2020 12:52 UTC <invalid> no
apiserver Nov 11, 2020 12:52 UTC <invalid> no
apiserver-etcd-client Nov 11, 2020 12:52 UTC <invalid> no
apiserver-kubelet-client Nov 11, 2020 12:52 UTC <invalid> no
controller-manager.conf Nov 11, 2020 12:52 UTC <invalid> no
etcd-healthcheck-client Nov 11, 2020 12:52 UTC <invalid> no
etcd-peer Nov 11, 2020 12:52 UTC <invalid> no
etcd-server Nov 11, 2020 12:52 UTC <invalid> no
front-proxy-client Nov 11, 2020 12:52 UTC <invalid> no
scheduler.conf Nov 11, 2020 12:52 UTC <invalid> no
# kubeadm alpha certs renew all
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
# kubeadm alpha certs check-expiration
CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
admin.conf Nov 17, 2021 05:49 UTC 364d no
apiserver Nov 17, 2021 05:49 UTC 364d no
apiserver-etcd-client Nov 17, 2021 05:49 UTC 364d no
apiserver-kubelet-client Nov 17, 2021 05:49 UTC 364d no
controller-manager.conf Nov 17, 2021 05:49 UTC 364d no
etcd-healthcheck-client Nov 17, 2021 05:49 UTC 364d no
etcd-peer Nov 17, 2021 05:49 UTC 364d no
etcd-server Nov 17, 2021 05:49 UTC 364d no
front-proxy-client Nov 17, 2021 05:49 UTC 364d no
scheduler.conf Nov 17, 2021 05:49 UTC 364d no
:~> mkdir -p $HOME/.kube
:~> sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
:~> sudo chown $(id -u):$(id -g) $HOME/.kube/config
:~> sudo systemctl daemon-reload
:~> sudo systemctl stop kubelet
:~> sudo systemctl start kubelet
:~> sudo systemctl enable kubelet
:~> sudo systemctl stop docker
:~> sudo systemctl start docker
:~> kubectl get pods
The connection to the server 10.xx.xx.74:6443 was refused - did you specify the right host or port?
# kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean"GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server 10.xx.xx.74:6443 was refused - did you specify the right host or port?
Kubectl 状态：
# systemctl status kubelet
kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2020-11-17 08:18:20 UTC; 1s ago
Docs: https://kubernetes.io/docs/
Process: 1452 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, sta
Main PID: 1452 (code=exited, status=255)
Nov 17 08:18:20 c536gocrb systemd[1]: Unit kubelet.service entered failed state.
Nov 17 08:18:20 c536gocrb systemd[1]: kubelet.service failed.
尝试将环境变量添加到管理员 10.kubeadm.conf
环境="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"
kubeadm 配置文件：
cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ==
server: https://xx.x.x.x.:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: system:node:cmaster
name: system:node:cmaster@kubernetes
current-context: system:node:cmaster@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:cmaster
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem