"x509: certificate signed by unknown authority" 当 运行 kubelet
"x509: certificate signed by unknown authority" when running kubelet
我正在尝试在 CoreOS beta (1192.2.0) 上使用 kubelet 1.4.5 安装 kubernetes。
我正在使用来自 https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic
的控制器和工作器安装脚本的略微修改版本
所以一般来说，我使用以下 bash 脚本在 Gentoo Linux 上创建了许可证：
#!/bin/bash
export MASTER_HOST=coreos-2.tux-in.com
export K8S_SERVICE_IP=10.3.0.1
export WORKER_IP=10.79.218.3
export WORKER_FQDN=coreos-3.tux-in.com
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"
openssl genrsa -out apiserver-key.pem 2048
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile openssl.cnf
openssl genrsa -out ${WORKER_FQDN}-worker-key.pem 2048
openssl req -new -key ${WORKER_FQDN}-worker-key.pem -out ${WORKER_FQDN}-worker.csr -subj "/CN=${WORKER_FQDN}" -config worker-openssl.cnf
openssl x509 -req -in ${WORKER_FQDN}-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ${WORKER_FQDN}-worker.pem -days 365 -extensions v3_req -extfile worker-openssl.cnf
openssl genrsa -out admin-key.pem 2048
openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=kube-admin"
openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 365
echo done
这是openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = coreos-2.tux-in.com
DNS.2 = coreos-3.tux-in.com
IP.1 = 10.3.0.1
IP.2 = 10.79.218.2
IP.3 = 10.79.218.3
这是我的 worker-openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.79.218.3
DNS.1 = coreos-3.tux-in.com
我的控制器机器是 coreos-2.tux-in.com 解析为 lan ip 10.79.218.2
我的工作机器是 coreos-3.tux-in.com 解析为 lan ip 10.79.218.3
它创建的许可证很好。但是当我使用它们并在主机上安装控制器脚本时，我看到当我 运行 journalctl -xef -u kubelet 并且我注意到以下消息时：
Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.805868 2018 event.go:208] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping)
Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.950827 2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461042 2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461340 2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.024366 2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.171170 2018 eviction_manager.go:162] eviction manager: unexpected err: failed GetNode: node '10.79.218.2' not found
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543619 2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543926 2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
kubelet documentation 说 --tls-cert-file 标志需要在证书后连接 CA。在你的情况下是 apiserver.pem:
--tls-cert-file File containing x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.
如果我没看错你的证书生成，apiserver.pem 不包含根证书。
我在 CoreOS 1192.2.0 上使用 kubelet 和 rkt。
这是我用来在 worker 上启动 kubelet 的单元：
[Unit]
Description=Kubelet via Hyperkube ACI
Requires=k8s-assets.target
After=k8s-assets.target
[Service]
EnvironmentFile=/etc/proxy.env
Environment="RKT_OPTS=--volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log"
Environment=KUBELET_VERSION=v1.4.0_coreos.0
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=https://10.203.69.108 \
--register-node=true \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--hostname-override=node2.my.domain \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
--tls-cert-file=/etc/kubernetes/ssl/worker.pem \
--tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
重要的是
--api-servers 必须指向 master
的 IP 地址
--tls-cert-file 必须指向工作者证书 public key
--tls-private-key-file 必须指向工作者证书私钥
--kubeconfig 必须指向有效的 kubeconfig 文件
这是我的 kubeconfig 文件（它包含已签署证书的 CA 的路径）：
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
certificate-authority: /etc/kubernetes/ssl/ca.pem
users:
- name: kubelet
user:
client-certificate: /etc/kubernetes/ssl/worker.pem
client-key: /etc/kubernetes/ssl/worker-key.pem
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
您的 OpenSSL 证书是 "self-signed":
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"
也就是说，您签署的是他们而不是受信任的 certificate authority。只要您妥善保管好私钥，它应该是完全没问题和安全的。
如果您希望它由证书颁发机构签名，您将需要生成 CSR（证书签名请求）。
一般来说，解决方案是创建另一个 etcd2 端口，连接到每台机器的环回设备并在 http 而不是 https 上工作。更多信息请访问
0. if your issue is :
: Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate"
1. look at your ca.crt
openssl x509 -noout -text -in ca.crt, you will find below info :
X509v3 Basic Constraints:
CA:FLASE
X509v3 Basic Constraints means :
"Basic Constraints" identifies if the subject of certificates is a CA who is allowed to issue child certificates. For a certificate that can be used to sign certificates, the info is in some sense duplicated: X509v3 Basic Constraints: CA: TRUE --- Can sign certificates.
you should modify it to CA:TRUE through vi openssl.conf
[ v3_ca ]
basicConstraints = CA:true
Regenerate your crts.
我正在尝试在 CoreOS beta (1192.2.0) 上使用 kubelet 1.4.5 安装 kubernetes。
我正在使用来自 https://github.com/coreos/coreos-kubernetes/tree/master/multi-node/generic
的控制器和工作器安装脚本的略微修改版本所以一般来说，我使用以下 bash 脚本在 Gentoo Linux 上创建了许可证：
#!/bin/bash
export MASTER_HOST=coreos-2.tux-in.com
export K8S_SERVICE_IP=10.3.0.1
export WORKER_IP=10.79.218.3
export WORKER_FQDN=coreos-3.tux-in.com
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"
openssl genrsa -out apiserver-key.pem 2048
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile openssl.cnf
openssl genrsa -out ${WORKER_FQDN}-worker-key.pem 2048
openssl req -new -key ${WORKER_FQDN}-worker-key.pem -out ${WORKER_FQDN}-worker.csr -subj "/CN=${WORKER_FQDN}" -config worker-openssl.cnf
openssl x509 -req -in ${WORKER_FQDN}-worker.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out ${WORKER_FQDN}-worker.pem -days 365 -extensions v3_req -extfile worker-openssl.cnf
openssl genrsa -out admin-key.pem 2048
openssl req -new -key admin-key.pem -out admin.csr -subj "/CN=kube-admin"
openssl x509 -req -in admin.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin.pem -days 365
echo done
这是openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = coreos-2.tux-in.com
DNS.2 = coreos-3.tux-in.com
IP.1 = 10.3.0.1
IP.2 = 10.79.218.2
IP.3 = 10.79.218.3
这是我的 worker-openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.79.218.3
DNS.1 = coreos-3.tux-in.com
我的控制器机器是 coreos-2.tux-in.com 解析为 lan ip 10.79.218.2
我的工作机器是 coreos-3.tux-in.com 解析为 lan ip 10.79.218.3
它创建的许可证很好。但是当我使用它们并在主机上安装控制器脚本时，我看到当我 运行 journalctl -xef -u kubelet 并且我注意到以下消息时：
Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.805868 2018 event.go:208] Unable to write event: 'x509: certificate signed by unknown authority' (may retry after sleeping)
Nov 08 21:24:06 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:06.950827 2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461042 2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:07 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:07.461340 2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.024366 2018 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://coreos-2.tux-in.com:443/api/v1/services?resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.171170 2018 eviction_manager.go:162] eviction manager: unexpected err: failed GetNode: node '10.79.218.2' not found
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543619 2018 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://coreos-2.tux-in.com:443/api/v1/nodes?fieldSelector=metadata.name%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
Nov 08 21:24:08 coreos-2.tux-in.com kubelet-wrapper[2018]: E1108 21:24:08.543926 2018 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://coreos-2.tux-in.com:443/api/v1/pods?fieldSelector=spec.nodeName%3D10.79.218.2&resourceVersion=0: x509: certificate signed by unknown authority
kubelet documentation 说 --tls-cert-file 标志需要在证书后连接 CA。在你的情况下是 apiserver.pem:
--tls-cert-file File containing x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir.
如果我没看错你的证书生成，apiserver.pem 不包含根证书。
我在 CoreOS 1192.2.0 上使用 kubelet 和 rkt。
这是我用来在 worker 上启动 kubelet 的单元：
[Unit]
Description=Kubelet via Hyperkube ACI
Requires=k8s-assets.target
After=k8s-assets.target
[Service]
EnvironmentFile=/etc/proxy.env
Environment="RKT_OPTS=--volume=resolv,kind=host,source=/etc/resolv.conf --mount volume=resolv,target=/etc/resolv.conf --volume var-log,kind=host,source=/var/log --mount volume=var-log,target=/var/log"
Environment=KUBELET_VERSION=v1.4.0_coreos.0
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=https://10.203.69.108 \
--register-node=true \
--allow-privileged=true \
--config=/etc/kubernetes/manifests \
--hostname-override=node2.my.domain \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml \
--tls-cert-file=/etc/kubernetes/ssl/worker.pem \
--tls-private-key-file=/etc/kubernetes/ssl/worker-key.pem
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
重要的是
--api-servers 必须指向 master
的 IP 地址--tls-cert-file 必须指向工作者证书 public key
--tls-private-key-file 必须指向工作者证书私钥
--kubeconfig 必须指向有效的 kubeconfig 文件
这是我的 kubeconfig 文件（它包含已签署证书的 CA 的路径）：
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
certificate-authority: /etc/kubernetes/ssl/ca.pem
users:
- name: kubelet
user:
client-certificate: /etc/kubernetes/ssl/worker.pem
client-key: /etc/kubernetes/ssl/worker-key.pem
contexts:
- context:
cluster: local
user: kubelet
name: kubelet-context
current-context: kubelet-context
您的 OpenSSL 证书是 "self-signed":
openssl genrsa -out ca-key.pem 2048
openssl req -x509 -new -nodes -key ca-key.pem -days 10000 -out ca.pem -subj "/CN=kube-ca"
也就是说，您签署的是他们而不是受信任的 certificate authority。只要您妥善保管好私钥，它应该是完全没问题和安全的。
如果您希望它由证书颁发机构签名，您将需要生成 CSR（证书签名请求）。
一般来说，解决方案是创建另一个 etcd2 端口，连接到每台机器的环回设备并在 http 而不是 https 上工作。更多信息请访问
0. if your issue is :
: Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate"
1. look at your ca.crt
openssl x509 -noout -text -in ca.crt, you will find below info :
X509v3 Basic Constraints:
CA:FLASE
X509v3 Basic Constraints means :
"Basic Constraints" identifies if the subject of certificates is a CA who is allowed to issue child certificates. For a certificate that can be used to sign certificates, the info is in some sense duplicated: X509v3 Basic Constraints: CA: TRUE --- Can sign certificates.
you should modify it to CA:TRUE through vi openssl.conf
[ v3_ca ]
basicConstraints = CA:true
Regenerate your crts.