使用自生成的 apiserver-kubelet-client.crt 证书初始化 K8S 集群时，一些 kubectl 命令不起作用
Some kubectl commands not working when K8S cluster is initialized with self-generated apiserver-kubelet-client.crt certificate
我需要为 Kubernetes 集群组件（apiserver、apiserver-kubelet-client、apiserver-etcd-client、front-proxy-client 等）生成自己的 SSL 证书。这样做的原因是因为这些证书的有效期默认设置为 1 年，我需要将有效期设置为超过一年，因为我的业务原因。当我生成自己的一组证书并初始化集群时，一切都运行良好——kube-system 命名空间中的 PODs 已启动，与 apiserver 的通信正常。但我遇到了一些命令，如 kubectl logs 或 kubectl port-forward 或 kubectl exec 停止工作并开始抛出以下错误：
kubectl logs <kube-apiserver-pod> -n kube-system
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <kube-apiserver-pod>))
或
kubectl exec -it <kube-apiserver-pod> -n kube-system sh
error: unable to upgrade connection: Unauthorized`
但是 docker exec 登录到 k8s_apiserver 容器的命令工作正常。
在调试过程中，我发现只有自生成的 apiserver-kubelet-client key/cert 文件会导致此集群行为。
Bellow 是我用来为 apiserver-kubelet-client 生成和使用我自己的 cert/key 对的过程。
我通过 运行 kubeadm init ...
/etc/kubernetes/pki 文件夹中
将 /etc/kubernetes/pki 文件夹备份到 /tmp/pki_k8s
用openssl打开apiserver-kubelet-client.crt检查所有设置的扩展，CN，O等
openssl x509 -noout -text -in /tmp/pki_k8s/apiserver-kubelet-client.crt
为了确保相同的扩展名和 CN,O 参数出现在我自己生成的证书中，我为扩展名创建了 .conf 文件，为 CN 和 O 创建了 .csr 文件
cd /tmp/pki_k8s/
cat <<-EOF_api_kubelet_client-ext > apiserver_kubelet_client-ext.conf
[ v3_ca ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
EOF_api_kubelet_client-ext
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"
终于生成了自己的apiserver-kubelet-client.crt。对于它的生成，我重用了现有的 apiserver-kubelet-client.key 和 ca.crt/ca.K8S 初始化生成的密钥
openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -out apiserver-kubelet-client.crt -extensions v3_ca -extfile apiserver_kubelet_client-ext.conf -days 3650
一旦我生成了自己的 apiserver-kubelet-client.crt，它覆盖了之前由 k8s 初始化脚本本身生成的 apiserver-kubelet-client.crt，我通过点击 kubeadm reset 来重置 kubernetes 集群。此已清除 /etc/kubernetes 文件夹
将所有证书从 /tmp/pki_k8s
/etc/kubernetes/pki
- 并重新初始化K8S集群
kubeadm init ...
在那期间，我看到 K8S 集群使用存储在 /etc/kubernetes/pki 中的现有证书进行设置。
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
之后，K8S 集群启动，我可以列出 pods、列出描述、进行部署等，但无法检查日志，如上所述执行命令。
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-kjkp9 1/1 Running 0 2m
coredns-78fcdf6894-q88lx 1/1 Running 0 2m
...
kubectl logs <apiserver_pod> -n kube-system -v 7
I0818 08:51:12.435494 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.436355 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.438413 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.447751 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.448109 12811 round_trippers.go:383] GET https://<HOST_IP>:6443/api/v1/namespaces/kube-system/pods/<apiserver_pod>
I0818 08:51:12.448126 12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.448135 12811 round_trippers.go:393] Accept: application/json, */*
I0818 08:51:12.448144 12811 round_trippers.go:393] User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.462931 12811 round_trippers.go:408] Response Status: 200 OK in 14 milliseconds
I0818 08:51:12.471316 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.471949 12811 round_trippers.go:383] GET https://<HOST_IP>:6443/api/v1/namespaces/kube-system/pods/<apiserver_pod>/log
I0818 08:51:12.471968 12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.471977 12811 round_trippers.go:393] Accept: application/json, */*
I0818 08:51:12.471985 12811 round_trippers.go:393] User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.475827 12811 round_trippers.go:408] Response Status: 401 Unauthorized in 3 milliseconds
I0818 08:51:12.476288 12811 helpers.go:201] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials ( pods/log <apiserver_pod>)",
"reason": "Unauthorized",
"details": {
"name": "<apiserver_pod>",
"kind": "pods/log"
},
"code": 401
}]
F0818 08:51:12.476325 12811 helpers.go:119] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <apiserver_pod>))
查看下面的 kubelet 服务文件：
[root@qa053 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="CA_CLIENT_CERT=--client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELE=--rotate-certificates=true"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CERTIFICATE_ARGS $CA_CLIENT_CERT
你有什么想法吗？ :)
谢谢
此致
我找到了它不起作用的原因。
创建 .csr 文件时我使用了这个：
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"
但是 -subj 中的格式错误导致从证书解析正确的 CN 时出现问题。而不是 "/O=system:masters,CN=kube-apiserver-kubelet-client" 它需要
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters/CN=kube-apiserver-kubelet-client"
两个 .csr 文件生成的证书在 -text 视图中看起来相同。但他们的行为不同。
我需要为 Kubernetes 集群组件（apiserver、apiserver-kubelet-client、apiserver-etcd-client、front-proxy-client 等）生成自己的 SSL 证书。这样做的原因是因为这些证书的有效期默认设置为 1 年，我需要将有效期设置为超过一年，因为我的业务原因。当我生成自己的一组证书并初始化集群时，一切都运行良好——kube-system 命名空间中的 PODs 已启动，与 apiserver 的通信正常。但我遇到了一些命令，如 kubectl logs 或 kubectl port-forward 或 kubectl exec 停止工作并开始抛出以下错误：
kubectl logs <kube-apiserver-pod> -n kube-system
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <kube-apiserver-pod>))
或
kubectl exec -it <kube-apiserver-pod> -n kube-system sh
error: unable to upgrade connection: Unauthorized`
但是 docker exec 登录到 k8s_apiserver 容器的命令工作正常。
在调试过程中，我发现只有自生成的 apiserver-kubelet-client key/cert 文件会导致此集群行为。
Bellow 是我用来为 apiserver-kubelet-client 生成和使用我自己的 cert/key 对的过程。
我通过 运行
kubeadm init ... 初始化 kubernetes 集群以将其自己的证书设置到 将
/etc/kubernetes/pki文件夹备份到/tmp/pki_k8s用openssl打开
apiserver-kubelet-client.crt检查所有设置的扩展，CN，O等openssl x509 -noout -text -in /tmp/pki_k8s/apiserver-kubelet-client.crt为了确保相同的扩展名和 CN,O 参数出现在我自己生成的证书中，我为扩展名创建了 .conf 文件，为 CN 和 O 创建了 .csr 文件
cd /tmp/pki_k8s/ cat <<-EOF_api_kubelet_client-ext > apiserver_kubelet_client-ext.conf [ v3_ca ] keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth EOF_api_kubelet_client-extopenssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"终于生成了自己的apiserver-kubelet-client.crt。对于它的生成，我重用了现有的 apiserver-kubelet-client.key 和 ca.crt/ca.K8S 初始化生成的密钥
openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial -sha256 -out apiserver-kubelet-client.crt -extensions v3_ca -extfile apiserver_kubelet_client-ext.conf -days 3650一旦我生成了自己的
apiserver-kubelet-client.crt，它覆盖了之前由 k8s 初始化脚本本身生成的apiserver-kubelet-client.crt，我通过点击kubeadm reset来重置 kubernetes 集群。此已清除 /etc/kubernetes 文件夹将所有证书从
/tmp/pki_k8s 复制到 - 并重新初始化K8S集群
kubeadm init ...
/etc/kubernetes/pki 文件夹中
/etc/kubernetes/pki
在那期间，我看到 K8S 集群使用存储在 /etc/kubernetes/pki 中的现有证书进行设置。
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Using the existing ca certificate and key.
[certificates] Using the existing apiserver certificate and key.
[certificates] Using the existing apiserver-kubelet-client certificate and key.
[certificates] Using the existing sa key.
[certificates] Using the existing front-proxy-ca certificate and key.
[certificates] Using the existing front-proxy-client certificate and key.
[certificates] Using the existing etcd/ca certificate and key.
[certificates] Using the existing etcd/server certificate and key.
[certificates] Using the existing etcd/peer certificate and key.
[certificates] Using the existing etcd/healthcheck-client certificate and key.
[certificates] Using the existing apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
之后，K8S 集群启动，我可以列出 pods、列出描述、进行部署等，但无法检查日志，如上所述执行命令。
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-kjkp9 1/1 Running 0 2m
coredns-78fcdf6894-q88lx 1/1 Running 0 2m
...
kubectl logs <apiserver_pod> -n kube-system -v 7
I0818 08:51:12.435494 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.436355 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.438413 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.447751 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.448109 12811 round_trippers.go:383] GET https://<HOST_IP>:6443/api/v1/namespaces/kube-system/pods/<apiserver_pod>
I0818 08:51:12.448126 12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.448135 12811 round_trippers.go:393] Accept: application/json, */*
I0818 08:51:12.448144 12811 round_trippers.go:393] User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.462931 12811 round_trippers.go:408] Response Status: 200 OK in 14 milliseconds
I0818 08:51:12.471316 12811 loader.go:359] Config loaded from file /root/.kube/config
I0818 08:51:12.471949 12811 round_trippers.go:383] GET https://<HOST_IP>:6443/api/v1/namespaces/kube-system/pods/<apiserver_pod>/log
I0818 08:51:12.471968 12811 round_trippers.go:390] Request Headers:
I0818 08:51:12.471977 12811 round_trippers.go:393] Accept: application/json, */*
I0818 08:51:12.471985 12811 round_trippers.go:393] User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f
I0818 08:51:12.475827 12811 round_trippers.go:408] Response Status: 401 Unauthorized in 3 milliseconds
I0818 08:51:12.476288 12811 helpers.go:201] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server has asked for the client to provide credentials ( pods/log <apiserver_pod>)",
"reason": "Unauthorized",
"details": {
"name": "<apiserver_pod>",
"kind": "pods/log"
},
"code": 401
}]
F0818 08:51:12.476325 12811 helpers.go:119] error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log <apiserver_pod>))
查看下面的 kubelet 服务文件：
[root@qa053 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
Environment="CA_CLIENT_CERT=--client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELE=--rotate-certificates=true"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CERTIFICATE_ARGS $CA_CLIENT_CERT
你有什么想法吗？ :) 谢谢
此致
我找到了它不起作用的原因。
创建 .csr 文件时我使用了这个：
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters,CN=kube-apiserver-kubelet-client"
但是 -subj 中的格式错误导致从证书解析正确的 CN 时出现问题。而不是 "/O=system:masters,CN=kube-apiserver-kubelet-client" 它需要
openssl req -new -key apiserver-kubelet-client.key -out apiserver-kubelet-client.csr -subj "/O=system:masters/CN=kube-apiserver-kubelet-client"
两个 .csr 文件生成的证书在 -text 视图中看起来相同。但他们的行为不同。