如何让kubelet通过HTTPS与apiserver通信？ v0.19
How to let kubelet communicate with apiserver by using HTTPS? v0.19
我在主节点 (core01) 上部署了 api 服务器，配置如下：
core01> /opt/bin/kube-apiserver \
--insecure_bind_address=127.0.0.1 \
--insecure_port=8080 \
--kubelet_port=10250 \
--etcd_servers=http://core01:2379,http://core02:2379,http://core03:2379 \
--service-cluster-ip-range=10.1.0.0/16 \
--allow_privileged=false \
--logtostderr=true \
--v=5 \
--tls-cert-file="/var/run/kubernetes/apiserver_36kr.pem" \
--tls-private-key-file="/var/run/kubernetes/apiserver_36kr.key" \
--client-ca-file="/var/run/kubernetes/cacert.pem" \
--kubelet-certificate-authority="/var/run/kubernetes/cacert.pem" \
--kubelet-client-certificate="/var/run/kubernetes/kubelet_36kr.pem" \
--kubelet-client-key="/var/run/kubernetes/kubelet_36kr.key"
在 minion 节点 (core02) 上，我可以从 HTTPS 调用 api:
core02> curl https://core01:6443/api/v1/nodes --cert /var/run/kubernetes/kubelet_36kr.pem --key /var/run/kubernetes/kubelet_36kr.key
> GET /api/v1/nodes HTTP/1.1
> Host: core01:6443
> User-Agent: curl/7.42.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 27 Jun 2015 15:33:50 GMT
< Content-Length: 1577
<
{
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/nodes",
"resourceVersion": "510078"
}, ....
但是，我无法在这个 minion 上启动 kubelet。它总是抱怨没有凭据。
我怎样才能让它发挥作用？有没有关于 master <-> minion 通信认证的文档？你能给我最好的做法吗？
仅供参考，命令如下：
core02> /opt/bin/kubelet \
--logtostderr=true \
--v=0 \
--api_servers=https://core01:6443 \
--address=127.0.0.1 \
--port=10250 \
--allow-privileged=false \
--tls-cert-file="/var/run/kubernetes/kubelet_36kr.pem" \
--tls-private-key-file="/var/run/kubernetes/kubelet_36kr.key"
kubelet 日志如下：
W0627 23:34:03.646311 3004 server.go:460] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0627 23:34:03.646520 3004 server.go:422] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0627 23:34:03.646710 3004 manager.go:127] cAdvisor running in container: "/system.slice/sshd.service"
I0627 23:34:03.647292 3004 fs.go:93] Filesystem partitions: map[/dev/sda9:{mountpoint:/ major:0 minor:30} /dev/sda4:{mountpoint:/usr major:8 minor:4} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6}]
I0627 23:34:03.648234 3004 manager.go:156] Machine: {NumCores:1 CpuFrequency:2399996 MemoryCapacity:1046294528 MachineID:29f94a4fad8b31668bd219ca511bdeb0 SystemUUID:4F4AF929-8BAD-6631-8BD2-19CA511BDEB0 BootID:fa1bea28-675e-4989-ad86-00797721a794 Filesystems:[{Device:/dev/sda9 Capacity:18987593728} {Device:/dev/sda4 Capacity:1031946240} {Device:/dev/sda6 Capacity:113229824}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:21474836480 Scheduler:cfq} 8:16:{Name:sdb Major:8 Minor:16 Size:1073741824 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:52:54:71:f6:fc:b8 Speed:0 Mtu:1500} {Name:flannel0 MacAddress: Speed:10 Mtu:1472}] Topology:[{Id:0 Memory:1046294528 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}]}
I0627 23:34:03.649934 3004 manager.go:163] Version: {KernelVersion:4.0.5 ContainerOsVersion:CoreOS 695.2.0 DockerVersion:1.6.2 CadvisorVersion:0.15.1}
I0627 23:34:03.651758 3004 plugins.go:69] No cloud provider specified.
I0627 23:34:03.651855 3004 docker.go:289] Connecting to docker on unix:///var/run/docker.sock
I0627 23:34:03.652877 3004 server.go:659] Watching apiserver
E0627 23:34:03.748954 3004 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)
E0627 23:34:03.750157 3004 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)
E0627 23:34:03.751666 3004 reflector.go:136] Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
I0627 23:34:03.758158 3004 plugins.go:56] Registering credential provider: .dockercfg
I0627 23:34:03.856215 3004 server.go:621] Started kubelet
E0627 23:34:03.858346 3004 kubelet.go:662] Image garbage collection failed: unable to find data for container /
I0627 23:34:03.869739 3004 kubelet.go:682] Running in container "/kubelet"
I0627 23:34:03.869755 3004 server.go:63] Starting to listen on 127.0.0.1:10250
E0627 23:34:03.899877 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba23275ceda25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"starting", Message:"Starting kubelet.", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
I0627 23:34:04.021297 3004 factory.go:226] System is using systemd
I0627 23:34:04.021790 3004 factory.go:234] Registering Docker factory
I0627 23:34:04.022241 3004 factory.go:89] Registering Raw factory
I0627 23:34:04.144065 3004 manager.go:946] Started watching for new ooms in manager
I0627 23:34:04.144655 3004 oomparser.go:183] oomparser using systemd
I0627 23:34:04.145379 3004 manager.go:243] Starting recovery of all containers
I0627 23:34:04.293020 3004 manager.go:248] Recovery completed
I0627 23:34:04.343829 3004 status_manager.go:56] Starting to sync pod status with apiserver
I0627 23:34:04.343928 3004 kubelet.go:1683] Starting kubelet main sync loop.
E0627 23:34:04.457765 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232995c8213", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
E0627 23:34:04.659874 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232a599cf8c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
kubelet 日志文件的前两行实际上指出了根本问题——您没有为 kubelet 连接到 master 指定任何客户端凭据。
kubelet 的--tls-cert-file 和--tls-private-key-file 参数用于配置kubelet 上的http 服务器（如果未指定，kubelet will generate a self-signed certificate for its https endpoint）。此证书/密钥对不用作提供给主服务器进行身份验证的客户端证书。
要指定凭据，有两个选项：kubeconfig 文件和 kubernetes_auth 文件。后者已被弃用，因此我建议使用 kubeconfig 文件。
在 kubeconfig 文件中，您需要指定 kubelet 应向 apiserver 提供的承载令牌或客户端证书。您还可以为 apiserver 指定 CA 证书（如果您希望连接安全）或告诉 kubelet 跳过检查 apiserver 提供的证书。由于您有 apiserver 的证书，我建议将 CA 证书添加到 kubeconfig 文件中。
kubeconfig 文件应该如下所示：
apiVersion: v1
kind: Config
users:
- name: kubelet
user:
client-certificate-data: <base64-encoded-cert>
client-key-data: <base64-encoded-key>
clusters:
- name: local
cluster:
certificate-authority-data: <base64-encoded-ca-cert>
contexts:
- context:
cluster: local
user: kubelet
name: service-account-context
current-context: service-account-context
要生成 base64 编码的客户端证书，您应该能够 运行 类似 cat /var/run/kubernetes/kubelet_36kr.pem | base64 的东西。如果手边没有 CA 证书，可以将 certificate-authority-data: <base64-encoded-ca-cert> 替换为 insecure-skip-tls-verify: true。
如果您将此文件放在 /var/lib/kubelet/kubeconfig，它应该会被自动拾取。否则，您可以使用 --kubeconfig 参数来指定自定义位置。
正如Robert Bailey所说，主要问题与连接主服务器的客户端凭据有关("Could not load kubeconfig file /var/lib/kubelet/kubeconfig...")。
我没有手动创建 kubeconfig 文件，而是选择使用 kubectl 工具生成它。
示例来自 docs：
$ kubectl config set-credentials myuser --username=myusername --password=mypassword
$ kubectl config set-cluster local-server --server=http://localhost:8080
$ kubectl config set-context default-context --cluster=local-server --user=myuser
$ kubectl config use-context default-context
$ kubectl config set contexts.default-context.namespace mynamespace
这些命令将在 ~/.kube/config
中生成一个配置文件
检查结果：
$ kubectl config view
然后我在 /var/lib/kubelet（默认位置）内为我的配置文件创建了一个符号 link：
$ cd /var/lib/kubelet
$ sudo ln -s ~/.kube/config kubeconfig
这对我有用。希望对你也有用。
全部归功于jnoller as he specifies the below commands on this issue。
他刚刚打错了，因为他跑了两次kubectl config set-credentials
这类似于 Robert Bailey 已接受的答案，除了您不需要任何 base64 并且它使编写脚本变得容易。
kubectl config set-cluster default-cluster --server=https://${MASTER} \
--certificate-authority=/path/to/ca.pem
kubectl config set-credentials default-admin \
--certificate-authority=/path/to/ca.pem \
--client-key=/path/to/admin-key.pem \
--client-certificate=/path/to/admin.pem
kubectl config set-context default-system --cluster=default-cluster --user=default-admin
kubectl config use-context default-system
在 ~/.kube/config 中生成的结果配置如下所示：
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/certs/ca.crt
server: https://kubernetesmaster
name: default-cluster
contexts:
- context:
cluster: default-cluster
user: default-admin
name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
user:
client-certificate: /etc/kubernetes/certs/server.crt
client-key: /etc/kubernetes/certs/server.key
我在主节点 (core01) 上部署了 api 服务器，配置如下：
core01> /opt/bin/kube-apiserver \
--insecure_bind_address=127.0.0.1 \
--insecure_port=8080 \
--kubelet_port=10250 \
--etcd_servers=http://core01:2379,http://core02:2379,http://core03:2379 \
--service-cluster-ip-range=10.1.0.0/16 \
--allow_privileged=false \
--logtostderr=true \
--v=5 \
--tls-cert-file="/var/run/kubernetes/apiserver_36kr.pem" \
--tls-private-key-file="/var/run/kubernetes/apiserver_36kr.key" \
--client-ca-file="/var/run/kubernetes/cacert.pem" \
--kubelet-certificate-authority="/var/run/kubernetes/cacert.pem" \
--kubelet-client-certificate="/var/run/kubernetes/kubelet_36kr.pem" \
--kubelet-client-key="/var/run/kubernetes/kubelet_36kr.key"
在 minion 节点 (core02) 上，我可以从 HTTPS 调用 api:
core02> curl https://core01:6443/api/v1/nodes --cert /var/run/kubernetes/kubelet_36kr.pem --key /var/run/kubernetes/kubelet_36kr.key
> GET /api/v1/nodes HTTP/1.1
> Host: core01:6443
> User-Agent: curl/7.42.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 27 Jun 2015 15:33:50 GMT
< Content-Length: 1577
<
{
"kind": "NodeList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/nodes",
"resourceVersion": "510078"
}, ....
但是，我无法在这个 minion 上启动 kubelet。它总是抱怨没有凭据。
我怎样才能让它发挥作用？有没有关于 master <-> minion 通信认证的文档？你能给我最好的做法吗？
仅供参考，命令如下：
core02> /opt/bin/kubelet \
--logtostderr=true \
--v=0 \
--api_servers=https://core01:6443 \
--address=127.0.0.1 \
--port=10250 \
--allow-privileged=false \
--tls-cert-file="/var/run/kubernetes/kubelet_36kr.pem" \
--tls-private-key-file="/var/run/kubernetes/kubelet_36kr.key"
kubelet 日志如下：
W0627 23:34:03.646311 3004 server.go:460] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W0627 23:34:03.646520 3004 server.go:422] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I0627 23:34:03.646710 3004 manager.go:127] cAdvisor running in container: "/system.slice/sshd.service"
I0627 23:34:03.647292 3004 fs.go:93] Filesystem partitions: map[/dev/sda9:{mountpoint:/ major:0 minor:30} /dev/sda4:{mountpoint:/usr major:8 minor:4} /dev/sda6:{mountpoint:/usr/share/oem major:8 minor:6}]
I0627 23:34:03.648234 3004 manager.go:156] Machine: {NumCores:1 CpuFrequency:2399996 MemoryCapacity:1046294528 MachineID:29f94a4fad8b31668bd219ca511bdeb0 SystemUUID:4F4AF929-8BAD-6631-8BD2-19CA511BDEB0 BootID:fa1bea28-675e-4989-ad86-00797721a794 Filesystems:[{Device:/dev/sda9 Capacity:18987593728} {Device:/dev/sda4 Capacity:1031946240} {Device:/dev/sda6 Capacity:113229824}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:21474836480 Scheduler:cfq} 8:16:{Name:sdb Major:8 Minor:16 Size:1073741824 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:52:54:71:f6:fc:b8 Speed:0 Mtu:1500} {Name:flannel0 MacAddress: Speed:10 Mtu:1472}] Topology:[{Id:0 Memory:1046294528 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[]}]}
I0627 23:34:03.649934 3004 manager.go:163] Version: {KernelVersion:4.0.5 ContainerOsVersion:CoreOS 695.2.0 DockerVersion:1.6.2 CadvisorVersion:0.15.1}
I0627 23:34:03.651758 3004 plugins.go:69] No cloud provider specified.
I0627 23:34:03.651855 3004 docker.go:289] Connecting to docker on unix:///var/run/docker.sock
I0627 23:34:03.652877 3004 server.go:659] Watching apiserver
E0627 23:34:03.748954 3004 reflector.go:136] Failed to list *api.Pod: the server has asked for the client to provide credentials (get pods)
E0627 23:34:03.750157 3004 reflector.go:136] Failed to list *api.Node: the server has asked for the client to provide credentials (get nodes)
E0627 23:34:03.751666 3004 reflector.go:136] Failed to list *api.Service: the server has asked for the client to provide credentials (get services)
I0627 23:34:03.758158 3004 plugins.go:56] Registering credential provider: .dockercfg
I0627 23:34:03.856215 3004 server.go:621] Started kubelet
E0627 23:34:03.858346 3004 kubelet.go:662] Image garbage collection failed: unable to find data for container /
I0627 23:34:03.869739 3004 kubelet.go:682] Running in container "/kubelet"
I0627 23:34:03.869755 3004 server.go:63] Starting to listen on 127.0.0.1:10250
E0627 23:34:03.899877 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba23275ceda25", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"starting", Message:"Starting kubelet.", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016043, nsec:856189989, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
I0627 23:34:04.021297 3004 factory.go:226] System is using systemd
I0627 23:34:04.021790 3004 factory.go:234] Registering Docker factory
I0627 23:34:04.022241 3004 factory.go:89] Registering Raw factory
I0627 23:34:04.144065 3004 manager.go:946] Started watching for new ooms in manager
I0627 23:34:04.144655 3004 oomparser.go:183] oomparser using systemd
I0627 23:34:04.145379 3004 manager.go:243] Starting recovery of all containers
I0627 23:34:04.293020 3004 manager.go:248] Recovery completed
I0627 23:34:04.343829 3004 status_manager.go:56] Starting to sync pod status with apiserver
I0627 23:34:04.343928 3004 kubelet.go:1683] Starting kubelet main sync loop.
E0627 23:34:04.457765 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232995c8213", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:452676115, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
E0627 23:34:04.659874 3004 event.go:185] Server rejected event '&api.Event{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"core02.13eba232a599cf8c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*util.Time)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil)}, InvolvedObject:api.ObjectReference{Kind:"Node", Namespace:"", Name:"core02", UID:"core02", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeReady", Message:"Node core02 status is now: NodeReady", Source:api.EventSource{Component:"kubelet", Host:"core02"}, FirstTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, LastTimestamp:util.Time{Time:time.Time{sec:63571016044, nsec:658020236, loc:(*time.Location)(0x1ba6120)}}, Count:1}': 'the server has asked for the client to provide credentials (post events)' (will not retry!)
kubelet 日志文件的前两行实际上指出了根本问题——您没有为 kubelet 连接到 master 指定任何客户端凭据。
kubelet 的--tls-cert-file 和--tls-private-key-file 参数用于配置kubelet 上的http 服务器（如果未指定，kubelet will generate a self-signed certificate for its https endpoint）。此证书/密钥对不用作提供给主服务器进行身份验证的客户端证书。
要指定凭据，有两个选项：kubeconfig 文件和 kubernetes_auth 文件。后者已被弃用，因此我建议使用 kubeconfig 文件。
在 kubeconfig 文件中，您需要指定 kubelet 应向 apiserver 提供的承载令牌或客户端证书。您还可以为 apiserver 指定 CA 证书（如果您希望连接安全）或告诉 kubelet 跳过检查 apiserver 提供的证书。由于您有 apiserver 的证书，我建议将 CA 证书添加到 kubeconfig 文件中。
kubeconfig 文件应该如下所示：
apiVersion: v1
kind: Config
users:
- name: kubelet
user:
client-certificate-data: <base64-encoded-cert>
client-key-data: <base64-encoded-key>
clusters:
- name: local
cluster:
certificate-authority-data: <base64-encoded-ca-cert>
contexts:
- context:
cluster: local
user: kubelet
name: service-account-context
current-context: service-account-context
要生成 base64 编码的客户端证书，您应该能够 运行 类似 cat /var/run/kubernetes/kubelet_36kr.pem | base64 的东西。如果手边没有 CA 证书，可以将 certificate-authority-data: <base64-encoded-ca-cert> 替换为 insecure-skip-tls-verify: true。
如果您将此文件放在 /var/lib/kubelet/kubeconfig，它应该会被自动拾取。否则，您可以使用 --kubeconfig 参数来指定自定义位置。
正如Robert Bailey所说，主要问题与连接主服务器的客户端凭据有关("Could not load kubeconfig file /var/lib/kubelet/kubeconfig...")。 我没有手动创建 kubeconfig 文件，而是选择使用 kubectl 工具生成它。
示例来自 docs：
$ kubectl config set-credentials myuser --username=myusername --password=mypassword
$ kubectl config set-cluster local-server --server=http://localhost:8080
$ kubectl config set-context default-context --cluster=local-server --user=myuser
$ kubectl config use-context default-context
$ kubectl config set contexts.default-context.namespace mynamespace
这些命令将在 ~/.kube/config
中生成一个配置文件检查结果：
$ kubectl config view
然后我在 /var/lib/kubelet（默认位置）内为我的配置文件创建了一个符号 link：
$ cd /var/lib/kubelet
$ sudo ln -s ~/.kube/config kubeconfig
这对我有用。希望对你也有用。
全部归功于jnoller as he specifies the below commands on this issue。
他刚刚打错了，因为他跑了两次kubectl config set-credentials
这类似于 Robert Bailey 已接受的答案，除了您不需要任何 base64 并且它使编写脚本变得容易。
kubectl config set-cluster default-cluster --server=https://${MASTER} \
--certificate-authority=/path/to/ca.pem
kubectl config set-credentials default-admin \
--certificate-authority=/path/to/ca.pem \
--client-key=/path/to/admin-key.pem \
--client-certificate=/path/to/admin.pem
kubectl config set-context default-system --cluster=default-cluster --user=default-admin
kubectl config use-context default-system
在 ~/.kube/config 中生成的结果配置如下所示：
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/certs/ca.crt
server: https://kubernetesmaster
name: default-cluster
contexts:
- context:
cluster: default-cluster
user: default-admin
name: default-system
current-context: default-system
kind: Config
preferences: {}
users:
- name: default-admin
user:
client-certificate: /etc/kubernetes/certs/server.crt
client-key: /etc/kubernetes/certs/server.key