无法在 Kubernetes 中创建 Secret：输入时有非法的 base64 数据
Can't create Secret in Kubernetes: illegal base64 data at input
我想为我的 kubernetes 集群创建一个秘密。所以我编写了以下 dummy-secret.yaml 文件：
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
当我 运行 kubectl create -f dummy-secret.yaml 我收到以下消息：
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret: Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...
不确定为什么会这样。
据我了解，我需要对 yaml 文件中 data 键下的所有值进行编码。所以我做了base64编码，但是kubernetes还是没有像我预期的那样处理yaml secret文件
更新：
我使用此命令对 mac 上的 data 值进行编码：
echo -n 'mega_secret_key' | openssl base64
我从你的编码数据中得到了解码值“mega_secret_key”和“really_secret_value1”。似乎它们没有以正确的方式编码。因此，以正确的方式编码您的数据：
$ echo "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5Cg==
$ echo "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTEK
然后检查编码是否正确：
$ echo "bWVnYV9zZWNyZXRfa2V5Cg==" | base64 -d
mega_secret_key
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTEK" | base64 -d
really_secret_value1
所以他们没事。现在在您的 dummy-secret.yaml:
中使用它们
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK
和运行 $ kubectl create -f dummy-secret.yaml.
2022 年 2 月 11 日更新：
较新版本的 Kubernetes 支持可选的 stringData 属性，其中可以在不解码的情况下提供针对任何密钥的值。
All key-value pairs in the stringData field are internally merged into the data field. If a key appears in both the data and the stringData field, the value specified in the stringData field takes precedence.
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
stringData:
API_KEY: mega_secret_key
API_SECRET: really_secret_value1
更新：
如果您在 运行 宁 $ echo "some_text" 时使用 -n 标志，它将 trim 您正在打印的字符串的尾部 \n（换行符） .
$ echo "some_text"
some_text
$ echo -n "some_text"
some_text⏎
试试吧，
# first encode
$ echo -n "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5
$ echo -n "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTE=
# then decode and check whether newline is stripped
$ echo "bWVnYV9zZWNyZXRfa2V5" | base64 -d
mega_secret_key⏎
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTE=" | base64 -d
really_secret_value1⏎
您可以在您的秘密中使用这些新的（没有换行符）解码数据。那也应该没问题。
$ cat - <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
EOF
secret/dummy-secret created
At the time of update, my kubernetes version is,
Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} Server
Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3",
GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213",
GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z",
GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} ```
看起来您的错误消息发生在另一个 dummy-secret.yaml。
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: af76fsdK_cQ89_Hj1Aq
API_SECRET: bsdfmkwegwegwe
然后：
$ kubectl create -f s.yaml
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...
如果我使用你的原件就可以正常工作：
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
然后：
$ kubectl create -f dummy-secret.yaml
secret/dummy-secret created
我使用的是以下版本：
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
尝试以错误的方式删除换行符时也可能发生这种情况（正确的方法是删除后缀 "Cg=="）。
我使用了来自 cli 的 base64，尽管有一些解决方法可以避免 NL，比如
https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225334
它们在 MacOS 中不起作用，我发现像这样使用 python 更简单
import base64
data = "abc123!?$*&()'-=@~"
# Standard Base64 Encoding
encodedBytes = base64.b64encode(data.encode("utf-8"))
encodedStr = str(encodedBytes, "utf-8")
或者仍然使用纯粹的基于bash的解决方案：
echo 'secret_to_encode' | tr -d \n | base64
过了一会儿我想 return 回到这个问题并留下一个参考官方 kubernetes 的答案 docs:
echo -n 'admin' | base64
YWRtaW4=
echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm
请特别注意-n，因为它保证在解码后您的密钥不会包含'new line symbol'。
这已得到解答，但为了将来参考，无需使用 stringData 而不是 data 字段对字符串进行编码，如下所示：
#secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
stringData:
API_KEY: "STRING_IN_CLEAR_TEXT"
API_SECRET: "STRING_IN_CLEAR_TEXT"
在我的例子中，我忘记了为一个键转换一个值。
它可以是这样的:)。
检查您的价值观
进行备份
sudo mkdir backup
sudo cp -R /etc/kubernetes backup/
sudo tar -cvzf backup/pki_backup_`hostname`-`date +%Y%m%d`.tar.gz backup/kubernetes/
查看目录 /etc/kubernetes/
ls -l
total 80
-rw------- 1 root root 5440 Mar 3 13:21 admin.conf
drwxr-xr-x 2 root root 4096 Aug 17 2020 audit-policy
-rw-r--r-- 1 root root 368 Mar 4 2020 calico-config.yml
-rw-r--r-- 1 root root 270 Mar 4 2020 calico-crb.yml
-rw-r--r-- 1 root root 341 Mar 4 2020 calico-cr.yml
-rw-r--r-- 1 root root 147 Mar 4 2020 calico-node-sa.yml
-rw-r--r-- 1 root root 6363 Mar 4 2020 calico-node.yml
-rw------- 1 root root 5472 Mar 3 13:21 controller-manager.conf
-rw-r--r-- 1 root root 3041 Aug 14 2020 kubeadm-config.v1alpha3.yaml
-rw------- 1 root root 5548 Mar 3 13:21 kubelet.conf
-rw-r--r-- 1 root root 1751 Mar 4 2020 kubelet.env
drwxr-xr-x 2 kube root 4096 Aug 14 2020 manifests
lrwxrwxrwx 1 root root 28 Mar 4 2020 node-kubeconfig.yaml -> /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5420 Mar 3 13:21 scheduler.conf
drwxr-xr-x 3 kube root 4096 Mar 3 10:20 ssl
尝试找到 k8s 集群配置，在我的例子中是
kubeadm-config.v1alpha3.yaml
如果没有相同的文件，你可以生成
kubectl get cm kubeadm-config -n kube-system -o yaml > /etc/kubernetes/kubeadm-config.yaml
在我的例子中也没有文件夹 /etc/kubernetes/pki 并且存在 ~/ssl
我从 /etc/kubernetes/ssl
创建符号链接 /etc/kubernetes/pki
ln -s /etc/kubernetes/ssl /etc/kubernetes/pki
让重新生成证书
kubeadm alpha phase certs apiserver --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Using the existing apiserver certificate and key.
kubeadm alpha phase certs apiserver-kubelet-client
I0303 13:12:24.543254 40613 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing apiserver-kubelet-client certificate and key.
kubeadm alpha phase certs front-proxy-client
I0303 13:12:35.660672 40989 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing front-proxy-client certificate and key.
kubeadm alpha phase certs etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [prod-uct1-mvp-k8s-0001 localhost] and IPs [127.0.0.1 ::1]
kubeadm alpha phase certs etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Using the existing etcd/server certificate and key.
kubeadm alpha phase certs etcd-healthcheck-client --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/healthcheck-client certificate and key.
kubeadm alpha phase certs etcd-peer --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [product1-mvp-k8s-0001 localhost] and IPs [192.168.4.201 127.0.0.1 ::1]
检查真实证书
find /etc/kubernetes/pki/ -name '*.crt' -exec openssl x509 -text -noout -in {} \; | grep -A2 Validity
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 10:29:44 2030 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 3 10:07:29 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 3 10:07:52 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 3 10:06:48 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 10:29:44 2030 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 19:39:56 2022 GMT
--
Validity
Not Before: Mar 4 10:29:43 2020 GMT
Not After : Mar 2 10:29:43 2030 GMT
--
Validity
Not Before: Mar 4 10:29:43 2020 GMT
Not After : Mar 2 19:40:13 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 19:36:38 2022 GMT
下一步生成新的配置文件：admin.conf，controller-manager.conf，kubelet.conf，scheduler.conf，在第一步将旧文件移动到 /tmp
cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/
kubeadm alpha phase kubeconfig all --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"
重启主 kubelet 并在重启后检查状态
sudo systemctl stop kubelet; sudo docker stop $(docker ps -aq); sudo docker rm $(docker ps -aq); sudo systemctl start kubelet
systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-03-03 14:00:22 MSK; 10s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 52998 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
Main PID: 53001 (kubelet)
Memory: 51.2M
CGroup: /system.slice/kubelet.service
检查访问主节点
kubectl get nodes
kubectl get ns
NAME STATUS AGE
default Active 464d
product1-mvp Active 318d
infra-logging Active 315d
infra-nginx-ingress Active 386d
kube-public Active 464d
kube-system Active 464d
pg Active 318d
检查证书
notAfter=Mar 3 07:40:43 2022 GMT
对所有主节点重复此过程。
现在我们准备更新工作节点的证书。
第一步我们必须删除或移动 kubelet.conf
cd /etc/kubernetes/
mv kubelet.conf kubelet.conf_old
变更后bootstrap-kubelet.conf
**api版本：v1
集群：
- 集群：
证书颁发机构数据：| LS0tLS1CRUDJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ETX
服务器：https://192.168.4.201:6443
名称：产品1
上下文：
- 上下文：
集群：产品1
用户：tls-bootstrap-令牌用户
名称：tls-bootstrap-token-user@product1
当前上下文：tls-bootstrap-token-user@product1
种类：配置
偏好： {}
用户：
- 名称：tls-bootstrap-令牌用户
用户：
代币：fgz9qz.lujw0bwsdfhdsfjhgds**
其中：
certificate-authority-data – 根证书 PKI CA master，可能取自 /etc/kubernetes/kubelet.conf on master nodes
- server:https://192.168.4.201:6443-ipapiserver主节点虚拟平衡ip
令牌：fgz9qz.lujw0bwsdfhdsfjhgds - 令牌，在主节点上生成
kubeadm token create
重启后检查kubelet工作节点和工作节点是否回到集群：
systemctl restart kubelet
systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-03-03 14:06:33 MSK; 11s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 54615 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
Main PID: 54621 (kubelet)
Memory: 52.1M
CGroup: /system.slice/kubelet.service
检查更新证书可能是目录中的 ls
ls -las /var/lib/kubelet/pki/
total 24
4 -rw-------. 1 root root 1135 Mar 3 14:06 kubelet-client-2021-03-03-14-06-34.pem
0 lrwxrwxrwx. 1 root root 59 Mar 3 14:06 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-03-03-14-06-34.pem
4 -rw-r--r--. 1 root root 2267 Mar 2 10:40 kubelet.crt
4 -rw-------. 1 root root 1679 Mar 2 10:40 kubelet.key
所有工作节点重复此过程后。
我遇到了同样的问题。
在我的例子中，编码很好——我只是复制了没有 "=".
最后一个字符的输出
可能导致相同错误的其他情况：
base64 复制输出中错位的空格或断线。
未在机密中指定 type 字段。
我想为我的 kubernetes 集群创建一个秘密。所以我编写了以下 dummy-secret.yaml 文件：
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
当我 运行 kubectl create -f dummy-secret.yaml 我收到以下消息：
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret: Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...
不确定为什么会这样。
据我了解，我需要对 yaml 文件中 data 键下的所有值进行编码。所以我做了base64编码，但是kubernetes还是没有像我预期的那样处理yaml secret文件
更新：
我使用此命令对 mac 上的 data 值进行编码：
echo -n 'mega_secret_key' | openssl base64
我从你的编码数据中得到了解码值“mega_secret_key”和“really_secret_value1”。似乎它们没有以正确的方式编码。因此，以正确的方式编码您的数据：
$ echo "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5Cg==
$ echo "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTEK
然后检查编码是否正确：
$ echo "bWVnYV9zZWNyZXRfa2V5Cg==" | base64 -d
mega_secret_key
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTEK" | base64 -d
really_secret_value1
所以他们没事。现在在您的 dummy-secret.yaml:
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5Cg==
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTEK
和运行 $ kubectl create -f dummy-secret.yaml.
2022 年 2 月 11 日更新：
较新版本的 Kubernetes 支持可选的 stringData 属性，其中可以在不解码的情况下提供针对任何密钥的值。
All key-value pairs in the
stringDatafield are internally merged into thedatafield. If a key appears in both thedataand thestringDatafield, the value specified in thestringDatafield takes precedence.
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
stringData:
API_KEY: mega_secret_key
API_SECRET: really_secret_value1
更新：
如果您在 运行 宁 $ echo "some_text" 时使用 -n 标志，它将 trim 您正在打印的字符串的尾部 \n（换行符） .
$ echo "some_text"
some_text
$ echo -n "some_text"
some_text⏎
试试吧，
# first encode
$ echo -n "mega_secret_key" | base64
bWVnYV9zZWNyZXRfa2V5
$ echo -n "really_secret_value1" | base64
cmVhbGx5X3NlY3JldF92YWx1ZTE=
# then decode and check whether newline is stripped
$ echo "bWVnYV9zZWNyZXRfa2V5" | base64 -d
mega_secret_key⏎
$ echo "cmVhbGx5X3NlY3JldF92YWx1ZTE=" | base64 -d
really_secret_value1⏎
您可以在您的秘密中使用这些新的（没有换行符）解码数据。那也应该没问题。
$ cat - <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
EOF
secret/dummy-secret created
At the time of update, my kubernetes version is,
Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c1 2ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"l inux/amd64"} ```
看起来您的错误消息发生在另一个 dummy-secret.yaml。
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: af76fsdK_cQ89_Hj1Aq
API_SECRET: bsdfmkwegwegwe
然后：
$ kubectl create -f s.yaml
Error from server (BadRequest): error when creating "dummy-secret.yaml": Secret in version "v1" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 8, error found in #10 byte of ...|Q89_Hj1Aq","API_SECR|..., bigger context ...|sion":"v1","data":{"API_KEY":"af76fsdK_cQ89_Hj1Aq","API_SECRET":"bsdfmkwegwegwe"},"kind":"Secret","m|...
如果我使用你的原件就可以正常工作：
apiVersion: v1
kind: Secret
metadata:
name: dummy-secret
type: Opaque
data:
API_KEY: bWVnYV9zZWNyZXRfa2V5
API_SECRET: cmVhbGx5X3NlY3JldF92YWx1ZTE=
然后：
$ kubectl create -f dummy-secret.yaml
secret/dummy-secret created
我使用的是以下版本：
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
尝试以错误的方式删除换行符时也可能发生这种情况（正确的方法是删除后缀 "Cg=="）。
我使用了来自 cli 的 base64，尽管有一些解决方法可以避免 NL，比如
https://superuser.com/questions/1225134/why-does-the-base64-of-a-string-contain-n/1225334
它们在 MacOS 中不起作用，我发现像这样使用 python 更简单
import base64
data = "abc123!?$*&()'-=@~"
# Standard Base64 Encoding
encodedBytes = base64.b64encode(data.encode("utf-8"))
encodedStr = str(encodedBytes, "utf-8")
或者仍然使用纯粹的基于bash的解决方案：
echo 'secret_to_encode' | tr -d \n | base64
过了一会儿我想 return 回到这个问题并留下一个参考官方 kubernetes 的答案 docs:
echo -n 'admin' | base64
YWRtaW4=
echo -n '1f2d1e2e67df' | base64
MWYyZDFlMmU2N2Rm
请特别注意-n，因为它保证在解码后您的密钥不会包含'new line symbol'。
这已得到解答，但为了将来参考，无需使用 stringData 而不是 data 字段对字符串进行编码，如下所示：
#secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
stringData:
API_KEY: "STRING_IN_CLEAR_TEXT"
API_SECRET: "STRING_IN_CLEAR_TEXT"
在我的例子中，我忘记了为一个键转换一个值。 它可以是这样的:)。 检查您的价值观
进行备份
sudo mkdir backup
sudo cp -R /etc/kubernetes backup/
sudo tar -cvzf backup/pki_backup_`hostname`-`date +%Y%m%d`.tar.gz backup/kubernetes/
查看目录 /etc/kubernetes/
ls -l
total 80
-rw------- 1 root root 5440 Mar 3 13:21 admin.conf
drwxr-xr-x 2 root root 4096 Aug 17 2020 audit-policy
-rw-r--r-- 1 root root 368 Mar 4 2020 calico-config.yml
-rw-r--r-- 1 root root 270 Mar 4 2020 calico-crb.yml
-rw-r--r-- 1 root root 341 Mar 4 2020 calico-cr.yml
-rw-r--r-- 1 root root 147 Mar 4 2020 calico-node-sa.yml
-rw-r--r-- 1 root root 6363 Mar 4 2020 calico-node.yml
-rw------- 1 root root 5472 Mar 3 13:21 controller-manager.conf
-rw-r--r-- 1 root root 3041 Aug 14 2020 kubeadm-config.v1alpha3.yaml
-rw------- 1 root root 5548 Mar 3 13:21 kubelet.conf
-rw-r--r-- 1 root root 1751 Mar 4 2020 kubelet.env
drwxr-xr-x 2 kube root 4096 Aug 14 2020 manifests
lrwxrwxrwx 1 root root 28 Mar 4 2020 node-kubeconfig.yaml -> /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5420 Mar 3 13:21 scheduler.conf
drwxr-xr-x 3 kube root 4096 Mar 3 10:20 ssl
尝试找到 k8s 集群配置，在我的例子中是 kubeadm-config.v1alpha3.yaml 如果没有相同的文件，你可以生成
kubectl get cm kubeadm-config -n kube-system -o yaml > /etc/kubernetes/kubeadm-config.yaml
在我的例子中也没有文件夹 /etc/kubernetes/pki 并且存在 ~/ssl 我从 /etc/kubernetes/ssl
创建符号链接 /etc/kubernetes/pkiln -s /etc/kubernetes/ssl /etc/kubernetes/pki
让重新生成证书
kubeadm alpha phase certs apiserver --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Using the existing apiserver certificate and key.
kubeadm alpha phase certs apiserver-kubelet-client
I0303 13:12:24.543254 40613 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing apiserver-kubelet-client certificate and key.
kubeadm alpha phase certs front-proxy-client
I0303 13:12:35.660672 40989 version.go:236] remote version is much newer: v1.20.4; falling back to: stable-1.12
[certificates] Using the existing front-proxy-client certificate and key.
kubeadm alpha phase certs etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [prod-uct1-mvp-k8s-0001 localhost] and IPs [127.0.0.1 ::1]
kubeadm alpha phase certs etcd-server --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Using the existing etcd/server certificate and key.
kubeadm alpha phase certs etcd-healthcheck-client --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/healthcheck-client certificate and key.
kubeadm alpha phase certs etcd-peer --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [product1-mvp-k8s-0001 localhost] and IPs [192.168.4.201 127.0.0.1 ::1]
检查真实证书
find /etc/kubernetes/pki/ -name '*.crt' -exec openssl x509 -text -noout -in {} \; | grep -A2 Validity
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 10:29:44 2030 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 3 10:07:29 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 3 10:07:52 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 3 10:06:48 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 10:29:44 2030 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 19:39:56 2022 GMT
--
Validity
Not Before: Mar 4 10:29:43 2020 GMT
Not After : Mar 2 10:29:43 2030 GMT
--
Validity
Not Before: Mar 4 10:29:43 2020 GMT
Not After : Mar 2 19:40:13 2022 GMT
--
Validity
Not Before: Mar 4 10:29:44 2020 GMT
Not After : Mar 2 19:36:38 2022 GMT
下一步生成新的配置文件：admin.conf，controller-manager.conf，kubelet.conf，scheduler.conf，在第一步将旧文件移动到 /tmp
cd /etc/kubernetes/
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} /tmp/
kubeadm alpha phase kubeconfig all --config /etc/kubernetes/kubeadm-config.v1alpha3.yaml
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Using existing up-to-date KubeConfig file: "/etc/kubernetes/scheduler.conf"
重启主 kubelet 并在重启后检查状态
sudo systemctl stop kubelet; sudo docker stop $(docker ps -aq); sudo docker rm $(docker ps -aq); sudo systemctl start kubelet
systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-03-03 14:00:22 MSK; 10s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 52998 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
Main PID: 53001 (kubelet)
Memory: 51.2M
CGroup: /system.slice/kubelet.service
检查访问主节点
kubectl get nodes
kubectl get ns
NAME STATUS AGE
default Active 464d
product1-mvp Active 318d
infra-logging Active 315d
infra-nginx-ingress Active 386d
kube-public Active 464d
kube-system Active 464d
pg Active 318d
检查证书
notAfter=Mar 3 07:40:43 2022 GMT
对所有主节点重复此过程。
现在我们准备更新工作节点的证书。 第一步我们必须删除或移动 kubelet.conf
cd /etc/kubernetes/
mv kubelet.conf kubelet.conf_old
变更后bootstrap-kubelet.conf
**api版本：v1 集群：
- 集群： 证书颁发机构数据：| LS0tLS1CRUDJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01ETX 服务器：https://192.168.4.201:6443 名称：产品1 上下文：
- 上下文： 集群：产品1 用户：tls-bootstrap-令牌用户 名称：tls-bootstrap-token-user@product1 当前上下文：tls-bootstrap-token-user@product1 种类：配置 偏好： {} 用户：
- 名称：tls-bootstrap-令牌用户 用户： 代币：fgz9qz.lujw0bwsdfhdsfjhgds**
其中： certificate-authority-data – 根证书 PKI CA master，可能取自 /etc/kubernetes/kubelet.conf on master nodes
- server:https://192.168.4.201:6443-ipapiserver主节点虚拟平衡ip
令牌：fgz9qz.lujw0bwsdfhdsfjhgds - 令牌，在主节点上生成
kubeadm token create
重启后检查kubelet工作节点和工作节点是否回到集群：
systemctl restart kubelet
systemctl status kubelet -l
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2021-03-03 14:06:33 MSK; 11s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Process: 54615 ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volume-plugins (code=exited, status=0/SUCCESS)
Main PID: 54621 (kubelet)
Memory: 52.1M
CGroup: /system.slice/kubelet.service
检查更新证书可能是目录中的 ls
ls -las /var/lib/kubelet/pki/
total 24
4 -rw-------. 1 root root 1135 Mar 3 14:06 kubelet-client-2021-03-03-14-06-34.pem
0 lrwxrwxrwx. 1 root root 59 Mar 3 14:06 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-03-03-14-06-34.pem
4 -rw-r--r--. 1 root root 2267 Mar 2 10:40 kubelet.crt
4 -rw-------. 1 root root 1679 Mar 2 10:40 kubelet.key
所有工作节点重复此过程后。
我遇到了同样的问题。
在我的例子中，编码很好——我只是复制了没有 "=".
可能导致相同错误的其他情况：
base64 复制输出中错位的空格或断线。
未在机密中指定
type字段。