x509: 证书由未知授权机构 metrics-server 签名
x509: certificate signed by unknown authority metrics-server
我是 kubernetes 的新手，我终于明白了如何按照文档 kubernetes-sigs/metrics-server 启动指标服务器。如果其他人想知道您需要在主节点上部署并且集群中至少有一个工作人员。
所以我得到这个错误：
E0818 15:25:22.835094 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:<hostname-master>: unable to fetch metrics from Kubelet <hostname-master> (<hostname-master>): Get https://<hostname-master>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:<hostname-worker>: unable to fetch metrics from Kubelet <hostname-worker> (<hostname-worker>): Get https://<hostname-worker>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]
我正在使用我自己的 CA（非自签名）并且我修改了 components.yml 文件（示例）：
args:
- --cert-dir=/tmp/metricsServerCas
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
我知道我可以通过使用这个标志来禁用 tls --kubelet-insecure-tls我已经试过了。我想使用我自己的 CA 来提高安全性。
我看到其他许多相关问题（样本很少）例如：
x509 certificate signed by unknown authority- Kubernetes and
虽然我已经应用了 chown $HOME/.kube/config 我仍然看到这个错误。
我哪里错了？
更新： 在 worker 上我正在创建一个目录，例如/tmp/ca 并在目录中添加 ca 文件。
我还不太了解安装点，我认为我做错了什么。可以在此处找到图像的默认语法 kubernetes-sigs/metrics-server/v0.3.7（请参阅 components.yml 文件）。
我试图在我的工作器上创建一个目录，例如/tmp/ca 并且我修改了标志 --cert-dir=/tmp/ca 和 mountPath: /tmp/ca
当我部署文件时，例如：
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
我不断收到来自 metrics-server-xxxx 的错误：
panic: open /tmp/client-ca-file805316981: read-only file system
尽管我已授予对目录的完全访问权限，例如：
$ ls -la /tmp/ca
total 8
drwxr-xr-x. 2 user user 20 Aug 19 16:59 .
drwxrwxrwt. 18 root root 4096 Aug 19 17:34 ..
-rwxr-xr-x. 1 user user 1025 Aug 19 16:59 ca.crt
我不确定我哪里错了。
如何配置才能让其他人可以使用非自签名证书？我可以看到大多数人都在使用我想避免的非 SSL。
图像中我的参数示例：
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp/ca
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp/ca
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
更新 2： 添加从 Master 到 Worker 的 curl 命令，包括错误输出：
$ curl --cacert /etc/kubernetes/pki/ca.crt https://node_hostname:10250/stats/summary?only_cpu_and_memory=true
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
创建一个 configmap 来存储用于生成 kubelet 服务证书的 ca 证书。
kubectl -n kube-system create configmap ca --from-file=ca.crt=/etc/kubernetes/pki/ca.crt -o yaml
然后使用volumeMounts在metrics server pod中使用它
spec:
volumes:
- emptyDir: {}
name: tmp-dir
- configMap:
defaultMode: 420
name: ca
name: ca-dir
containers:
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-certificate-authority=/ca/ca.crt
- --kubelet-preferred-address-types=Hostname
volumeMounts:
- mountPath: /tmp
name: tmp-dir
- mountPath: /ca
name: ca-dir
您可以按照相同的方法使用 --tls-cert-file 和 --tls-private-key-file 来使用您自己的证书而不是自签名证书。
将此答案作为社区 Wiki 发布，以便在评论中发布解决方案时提供更好的可见性。
The version that I used before was 1.18.2 and metrics server v0.3.6. Deployment was through kubeadm. Yes all requirements was exactly as the metrics-server/requirements. The good news is that I got it running by upgrading my k8s version on 1.19.0 and using the latest version v0.3.7. It works with self signed certificates.
问题已通过升级解决：
Kubernetes: 1.18.2 -> 1.19.0Metrics-server: 0.3.6 -> 0.3.7
此升级允许 运行 metrics-server 启用 tls（self-signed 证书）。
在使用 tls 部署 metrics-server 时可以提供帮助的其他资源：
How to run metrics-server securely?
Suggested configuration:
- Cluster with RBAC enabled
- Kubelet read-only port port disabled
- Validate kubelet certificate by mounting CA file and providing --kubelet-certificate-authority flag to metrics server
- Avoid passing insecure flags to metrics server (--deprecated-kubelet-completely-insecure, --kubelet-insecure-tls)
- Consider using your own certificates (--tls-cert-file, --tls-private-key-file)
对于我在 EKS 上的朋友，请确保您设置了用户名（而不是像我一样设置为会话名称）：
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: {{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
error: You must be logged in to the server (Unauthorized)
robert ❱ 1 ❱ kubectl edit configmap -n kube-system aws-auth
configmap/aws-auth edited
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: literally_anything:{{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-3-103.us-west-2.compute.internal 341m 17% 1738Mi 52%
...
robert ❱ kubectl logs -n kube-system -l app.kubernetes.io/instance=metrics-server
E0407 22:34:45.879156 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:49.399854 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:50.691133 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:51.827629 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:07.288163 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:08.755492 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:09.801957 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:40:32.405458 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:43:09.791769 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:44:14.244221 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
robert ❱
我是 kubernetes 的新手，我终于明白了如何按照文档 kubernetes-sigs/metrics-server 启动指标服务器。如果其他人想知道您需要在主节点上部署并且集群中至少有一个工作人员。
所以我得到这个错误：
E0818 15:25:22.835094 1 manager.go:111] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:<hostname-master>: unable to fetch metrics from Kubelet <hostname-master> (<hostname-master>): Get https://<hostname-master>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:<hostname-worker>: unable to fetch metrics from Kubelet <hostname-worker> (<hostname-worker>): Get https://<hostname-worker>:10250/stats/summary?only_cpu_and_memory=true: x509: certificate signed by unknown authority]
我正在使用我自己的 CA（非自签名）并且我修改了 components.yml 文件（示例）：
args:
- --cert-dir=/tmp/metricsServerCas
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
我知道我可以通过使用这个标志来禁用 tls --kubelet-insecure-tls我已经试过了。我想使用我自己的 CA 来提高安全性。
我看到其他许多相关问题（样本很少）例如：
x509 certificate signed by unknown authority- Kubernetes and
虽然我已经应用了 chown $HOME/.kube/config 我仍然看到这个错误。
我哪里错了？
更新： 在 worker 上我正在创建一个目录，例如/tmp/ca 并在目录中添加 ca 文件。
我还不太了解安装点，我认为我做错了什么。可以在此处找到图像的默认语法 kubernetes-sigs/metrics-server/v0.3.7（请参阅 components.yml 文件）。
我试图在我的工作器上创建一个目录，例如/tmp/ca 并且我修改了标志 --cert-dir=/tmp/ca 和 mountPath: /tmp/ca
当我部署文件时，例如：
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
我不断收到来自 metrics-server-xxxx 的错误：
panic: open /tmp/client-ca-file805316981: read-only file system
尽管我已授予对目录的完全访问权限，例如：
$ ls -la /tmp/ca
total 8
drwxr-xr-x. 2 user user 20 Aug 19 16:59 .
drwxrwxrwt. 18 root root 4096 Aug 19 17:34 ..
-rwxr-xr-x. 1 user user 1025 Aug 19 16:59 ca.crt
我不确定我哪里错了。
如何配置才能让其他人可以使用非自签名证书？我可以看到大多数人都在使用我想避免的非 SSL。
图像中我的参数示例：
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp/ca
- --secure-port=4443
- --kubelet-preferred-address-types=Hostname
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp/ca
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
更新 2： 添加从 Master 到 Worker 的 curl 命令，包括错误输出：
$ curl --cacert /etc/kubernetes/pki/ca.crt https://node_hostname:10250/stats/summary?only_cpu_and_memory=true
curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
创建一个 configmap 来存储用于生成 kubelet 服务证书的 ca 证书。
kubectl -n kube-system create configmap ca --from-file=ca.crt=/etc/kubernetes/pki/ca.crt -o yaml
然后使用volumeMounts在metrics server pod中使用它
spec:
volumes:
- emptyDir: {}
name: tmp-dir
- configMap:
defaultMode: 420
name: ca
name: ca-dir
containers:
args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-certificate-authority=/ca/ca.crt
- --kubelet-preferred-address-types=Hostname
volumeMounts:
- mountPath: /tmp
name: tmp-dir
- mountPath: /ca
name: ca-dir
您可以按照相同的方法使用 --tls-cert-file 和 --tls-private-key-file 来使用您自己的证书而不是自签名证书。
将此答案作为社区 Wiki 发布，以便在评论中发布解决方案时提供更好的可见性。
The version that I used before was 1.18.2 and metrics server v0.3.6. Deployment was through kubeadm. Yes all requirements was exactly as the metrics-server/requirements. The good news is that I got it running by upgrading my k8s version on 1.19.0 and using the latest version v0.3.7. It works with self signed certificates.
问题已通过升级解决：
Kubernetes:1.18.2->1.19.0Metrics-server:0.3.6->0.3.7
此升级允许 运行 metrics-server 启用 tls（self-signed 证书）。
在使用 tls 部署 metrics-server 时可以提供帮助的其他资源：
How to run metrics-server securely? Suggested configuration:
- Cluster with RBAC enabled
- Kubelet read-only port port disabled
- Validate kubelet certificate by mounting CA file and providing --kubelet-certificate-authority flag to metrics server
- Avoid passing insecure flags to metrics server (--deprecated-kubelet-completely-insecure, --kubelet-insecure-tls)
- Consider using your own certificates (--tls-cert-file, --tls-private-key-file)
对于我在 EKS 上的朋友，请确保您设置了用户名（而不是像我一样设置为会话名称）：
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: {{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
error: You must be logged in to the server (Unauthorized)
robert ❱ 1 ❱ kubectl edit configmap -n kube-system aws-auth
configmap/aws-auth edited
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123546789012:role/MyTeamRole
username: literally_anything:{{SessionName}}
groups:
- system:masters
robert ❱ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
ip-10-0-3-103.us-west-2.compute.internal 341m 17% 1738Mi 52%
...
robert ❱ kubectl logs -n kube-system -l app.kubernetes.io/instance=metrics-server
E0407 22:34:45.879156 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:49.399854 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:50.691133 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:51.827629 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:07.288163 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:08.755492 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:09.801957 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:40:32.405458 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:43:09.791769 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:44:14.244221 1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
robert ❱