当 运行 Kubernetes 在本地通过 Docker 时，Master 未启动
Master not starting when running Kubernetes locally via Docker
我正在按照 Running Kubernetes locally via Docker 指南进行操作，但我无法让 master 正常启动。
第一步：运行etcd
docker run --net=host -d gcr.io/google_containers/etcd:2.0.9 /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data
etcd 容器似乎正常启动。 docker logs 没有看到任何错误，我最终在 4001 上监听了一个 etcd 进程。
第二步：运行大师
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock gcr.io/google_containers/hyperkube:v0.21.2 /hyperkube kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests
我相信这就是我的问题开始的地方。以下是 docker logs:
的输出
W1021 13:23:04.093281 1 server.go:259] failed to set oom_score_adj to -900: write /proc/self/oom_score_adj: permission denied
W1021 13:23:04.093426 1 server.go:462] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
W1021 13:23:04.093445 1 server.go:424] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
I1021 13:23:04.093503 1 server.go:271] Using root directory: /var/lib/kubelet
I1021 13:23:04.093519 1 plugins.go:69] No cloud provider specified.
I1021 13:23:04.093526 1 server.go:290] Successfully initialized cloud provider: "" from the config file: ""
I1021 13:23:05.126191 1 docker.go:289] Connecting to docker on unix:///var/run/docker.sock
I1021 13:23:05.126396 1 server.go:651] Adding manifest file: /etc/kubernetes/manifests
I1021 13:23:05.126409 1 file.go:47] Watching path "/etc/kubernetes/manifests"
I1021 13:23:05.126416 1 server.go:661] Watching apiserver
E1021 13:23:05.127148 1 reflector.go:136] Failed to list *api.Pod: Get http://localhost:8080/api/v1/pods?fieldSelector=spec.nodeName%3D127.0.0.1: dial tcp 127.0.0.1:8080: connection refused
E1021 13:23:05.127295 1 reflector.go:136] Failed to list *api.Service: Get http://localhost:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused
E1021 13:23:05.127336 1 reflector.go:136] Failed to list *api.Node: Get http://localhost:8080/api/v1/nodes?fieldSelector=metadata.name%3D127.0.0.1: dial tcp 127.0.0.1:8080: connection refused
I1021 13:23:05.343848 1 plugins.go:56] Registering credential provider: .dockercfg
W1021 13:23:05.394268 1 container_manager_linux.go:96] Memory limit 0 for container /docker-daemon is too small, reset it to 157286400
I1021 13:23:05.394284 1 container_manager_linux.go:100] Configure resource-only container /docker-daemon with memory limit: 157286400
I1021 13:23:05.395019 1 plugins.go:180] Loaded volume plugin "kubernetes.io/aws-ebs"
I1021 13:23:05.395040 1 plugins.go:180] Loaded volume plugin "kubernetes.io/empty-dir"
I1021 13:23:05.395052 1 plugins.go:180] Loaded volume plugin "empty"
I1021 13:23:05.395068 1 plugins.go:180] Loaded volume plugin "kubernetes.io/gce-pd"
I1021 13:23:05.395080 1 plugins.go:180] Loaded volume plugin "gce-pd"
I1021 13:23:05.395098 1 plugins.go:180] Loaded volume plugin "kubernetes.io/git-repo"
I1021 13:23:05.395112 1 plugins.go:180] Loaded volume plugin "git"
I1021 13:23:05.395124 1 plugins.go:180] Loaded volume plugin "kubernetes.io/host-path"
I1021 13:23:05.395136 1 plugins.go:180] Loaded volume plugin "kubernetes.io/nfs"
I1021 13:23:05.395147 1 plugins.go:180] Loaded volume plugin "kubernetes.io/secret"
I1021 13:23:05.395156 1 plugins.go:180] Loaded volume plugin "kubernetes.io/iscsi"
I1021 13:23:05.395166 1 plugins.go:180] Loaded volume plugin "kubernetes.io/glusterfs"
I1021 13:23:05.395178 1 plugins.go:180] Loaded volume plugin "kubernetes.io/persistent-claim"
I1021 13:23:05.395194 1 plugins.go:180] Loaded volume plugin "kubernetes.io/rbd"
I1021 13:23:05.395274 1 server.go:623] Started kubelet
I1021 13:23:05.395296 1 server.go:63] Starting to listen on 0.0.0.0:10250
I1021 13:23:05.395507 1 server.go:82] Starting to listen read-only on 0.0.0.0:10255
第三步：运行服务代理
docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v0.21.2 /hyperkube proxy --master=http://127.0.0.1:8080 --v=2
此步骤的 docker 日志包含与我在第二步中看到的类似的错误。
I1021 13:32:03.177004 1 server.go:88] Running in resource-only container "/kube-proxy"
I1021 13:32:03.177432 1 proxier.go:121] Setting proxy IP to 192.168.19.200 and initializing iptables
E1021 13:32:03.195731 1 api.go:108] Unable to load services: Get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused
E1021 13:32:03.195924 1 api.go:180] Unable to load endpoints: Get http://127.0.0.1:8080/api/v1/endpoints: dial tcp 127.0.0.1:8080: connection refused
docker ps 输出：
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
576d15c22537 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube proxy --m" About an hour ago Up About an hour high_pasteur
a98637c9d523 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube kubelet -" About an hour ago Up 34 minutes drunk_jones
618afb1de613 gcr.io/google_containers/etcd:2.0.9 "/usr/local/bin/etcd " 2 hours ago Up 2 hours high_yonath
第 2 步日志中的第一个错误让我相信该错误可能与 iptables 有关。
iptables -L 输出：
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
docker exec -ti a98637c9d523 cat /etc/kubernetes/manifests/master.json 输出：
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {"name":"k8s-master"},
"spec":{
"hostNetwork": true,
"containers":[
{
"name": "controller-manager",
"image": "gcr.io/google_containers/hyperkube:v1.0.6",
"command": [
"/hyperkube",
"controller-manager",
"--master=127.0.0.1:8080",
"--v=2"
]
},
{
"name": "apiserver",
"image": "gcr.io/google_containers/hyperkube:v1.0.6",
"command": [
"/hyperkube",
"apiserver",
"--portal-net=10.0.0.1/24",
"--address=127.0.0.1",
"--etcd-servers=http://127.0.0.1:4001",
"--cluster-name=kubernetes",
"--v=2"
]
},
{
"name": "scheduler",
"image": "gcr.io/google_containers/hyperkube:v1.0.6",
"command": [
"/hyperkube",
"scheduler",
"--master=127.0.0.1:8080",
"--v=2"
]
}
]
}
}
Docker 版本 1.8.3
内核版本 4.2.3
如有任何见解，我们将不胜感激。
可以先将docker版本降级到1.7.2吗？我用 docker 1.7.2 做了你上面所做的，一切正常。
$ curl 127.0.0.1:8080/
{
"paths": [
"/api",
"/api/v1",
"/api/v1beta3",
"/healthz",
"/healthz/ping",
"/logs/",
"/metrics",
"/resetMetrics",
"/swagger-ui/",
"/swaggerapi/",
"/ui/",
"/version"
]
}
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0141e596414c gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube proxy -- 15 minutes ago Up 15 minutes nostalgic_nobel
10634ce798e9 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube schedule 16 minutes ago Up 16 minutes k8s_scheduler.b725e775_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_43562383
5618a39eb11d gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube apiserve 16 minutes ago Up 16 minutes k8s_apiserver.70750283_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_e5d145be
25f336102b26 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube controll 16 minutes ago Up 16 minutes k8s_controller-manager.aad1ee8f_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_fe538b9b
7f1391840920 gcr.io/google_containers/pause:0.8.0 "/pause" 17 minutes ago Up 17 minutes k8s_POD.e4cc795_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_26fd84fd
a11715435f45 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube kubelet 17 minutes ago Up 17 minutes jovial_hodgkin
a882a1a4b917 gcr.io/google_containers/etcd:2.0.9 "/usr/local/bin/etcd 18 minutes ago Up 18 minutes adoring_hodgkin
docker 1.8.3 存在一些已知问题，尤其是 docker#17190. We had to workaround such issue through kubernetes#16052。但是这些变化并不是 Kubernetes 1.0 版本中特意挑选出来的。从您上面发布的输出中，我注意到没有暂停容器。您还可以 运行 docker ps -a 检查某些容器是否已死，然后将 docker logs <dead-container> 的输出复制并粘贴到此处？
我将提交一个问题以确保 Kubernetes 1.1 版本与 docker 1.8.3 一起正常工作。谢谢！
我正在按照 Running Kubernetes locally via Docker 指南进行操作，但我无法让 master 正常启动。
第一步：运行etcd
docker run --net=host -d gcr.io/google_containers/etcd:2.0.9 /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/data
etcd 容器似乎正常启动。 docker logs 没有看到任何错误，我最终在 4001 上监听了一个 etcd 进程。
第二步：运行大师
docker run --net=host -d -v /var/run/docker.sock:/var/run/docker.sock gcr.io/google_containers/hyperkube:v0.21.2 /hyperkube kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifests
我相信这就是我的问题开始的地方。以下是 docker logs:
W1021 13:23:04.093281 1 server.go:259] failed to set oom_score_adj to -900: write /proc/self/oom_score_adj: permission denied W1021 13:23:04.093426 1 server.go:462] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead. W1021 13:23:04.093445 1 server.go:424] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults. I1021 13:23:04.093503 1 server.go:271] Using root directory: /var/lib/kubelet I1021 13:23:04.093519 1 plugins.go:69] No cloud provider specified. I1021 13:23:04.093526 1 server.go:290] Successfully initialized cloud provider: "" from the config file: "" I1021 13:23:05.126191 1 docker.go:289] Connecting to docker on unix:///var/run/docker.sock I1021 13:23:05.126396 1 server.go:651] Adding manifest file: /etc/kubernetes/manifests I1021 13:23:05.126409 1 file.go:47] Watching path "/etc/kubernetes/manifests" I1021 13:23:05.126416 1 server.go:661] Watching apiserver E1021 13:23:05.127148 1 reflector.go:136] Failed to list *api.Pod: Get http://localhost:8080/api/v1/pods?fieldSelector=spec.nodeName%3D127.0.0.1: dial tcp 127.0.0.1:8080: connection refused E1021 13:23:05.127295 1 reflector.go:136] Failed to list *api.Service: Get http://localhost:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused E1021 13:23:05.127336 1 reflector.go:136] Failed to list *api.Node: Get http://localhost:8080/api/v1/nodes?fieldSelector=metadata.name%3D127.0.0.1: dial tcp 127.0.0.1:8080: connection refused I1021 13:23:05.343848 1 plugins.go:56] Registering credential provider: .dockercfg W1021 13:23:05.394268 1 container_manager_linux.go:96] Memory limit 0 for container /docker-daemon is too small, reset it to 157286400 I1021 13:23:05.394284 1 container_manager_linux.go:100] Configure resource-only container /docker-daemon with memory limit: 157286400 I1021 13:23:05.395019 1 plugins.go:180] Loaded volume plugin "kubernetes.io/aws-ebs" I1021 13:23:05.395040 1 plugins.go:180] Loaded volume plugin "kubernetes.io/empty-dir" I1021 13:23:05.395052 1 plugins.go:180] Loaded volume plugin "empty" I1021 13:23:05.395068 1 plugins.go:180] Loaded volume plugin "kubernetes.io/gce-pd" I1021 13:23:05.395080 1 plugins.go:180] Loaded volume plugin "gce-pd" I1021 13:23:05.395098 1 plugins.go:180] Loaded volume plugin "kubernetes.io/git-repo" I1021 13:23:05.395112 1 plugins.go:180] Loaded volume plugin "git" I1021 13:23:05.395124 1 plugins.go:180] Loaded volume plugin "kubernetes.io/host-path" I1021 13:23:05.395136 1 plugins.go:180] Loaded volume plugin "kubernetes.io/nfs" I1021 13:23:05.395147 1 plugins.go:180] Loaded volume plugin "kubernetes.io/secret" I1021 13:23:05.395156 1 plugins.go:180] Loaded volume plugin "kubernetes.io/iscsi" I1021 13:23:05.395166 1 plugins.go:180] Loaded volume plugin "kubernetes.io/glusterfs" I1021 13:23:05.395178 1 plugins.go:180] Loaded volume plugin "kubernetes.io/persistent-claim" I1021 13:23:05.395194 1 plugins.go:180] Loaded volume plugin "kubernetes.io/rbd" I1021 13:23:05.395274 1 server.go:623] Started kubelet I1021 13:23:05.395296 1 server.go:63] Starting to listen on 0.0.0.0:10250 I1021 13:23:05.395507 1 server.go:82] Starting to listen read-only on 0.0.0.0:10255
第三步：运行服务代理
docker run -d --net=host --privileged gcr.io/google_containers/hyperkube:v0.21.2 /hyperkube proxy --master=http://127.0.0.1:8080 --v=2
此步骤的 docker 日志包含与我在第二步中看到的类似的错误。
I1021 13:32:03.177004 1 server.go:88] Running in resource-only container "/kube-proxy" I1021 13:32:03.177432 1 proxier.go:121] Setting proxy IP to 192.168.19.200 and initializing iptables E1021 13:32:03.195731 1 api.go:108] Unable to load services: Get http://127.0.0.1:8080/api/v1/services: dial tcp 127.0.0.1:8080: connection refused E1021 13:32:03.195924 1 api.go:180] Unable to load endpoints: Get http://127.0.0.1:8080/api/v1/endpoints: dial tcp 127.0.0.1:8080: connection refused
docker ps 输出：
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 576d15c22537 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube proxy --m" About an hour ago Up About an hour high_pasteur a98637c9d523 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube kubelet -" About an hour ago Up 34 minutes drunk_jones 618afb1de613 gcr.io/google_containers/etcd:2.0.9 "/usr/local/bin/etcd " 2 hours ago Up 2 hours high_yonath
第 2 步日志中的第一个错误让我相信该错误可能与 iptables 有关。
iptables -L 输出：
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (1 references) target prot opt source destination
docker exec -ti a98637c9d523 cat /etc/kubernetes/manifests/master.json 输出：
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {"name":"k8s-master"},
"spec":{
"hostNetwork": true,
"containers":[
{
"name": "controller-manager",
"image": "gcr.io/google_containers/hyperkube:v1.0.6",
"command": [
"/hyperkube",
"controller-manager",
"--master=127.0.0.1:8080",
"--v=2"
]
},
{
"name": "apiserver",
"image": "gcr.io/google_containers/hyperkube:v1.0.6",
"command": [
"/hyperkube",
"apiserver",
"--portal-net=10.0.0.1/24",
"--address=127.0.0.1",
"--etcd-servers=http://127.0.0.1:4001",
"--cluster-name=kubernetes",
"--v=2"
]
},
{
"name": "scheduler",
"image": "gcr.io/google_containers/hyperkube:v1.0.6",
"command": [
"/hyperkube",
"scheduler",
"--master=127.0.0.1:8080",
"--v=2"
]
}
]
}
}
Docker 版本 1.8.3
内核版本 4.2.3
如有任何见解，我们将不胜感激。
可以先将docker版本降级到1.7.2吗？我用 docker 1.7.2 做了你上面所做的，一切正常。
$ curl 127.0.0.1:8080/
{
"paths": [
"/api",
"/api/v1",
"/api/v1beta3",
"/healthz",
"/healthz/ping",
"/logs/",
"/metrics",
"/resetMetrics",
"/swagger-ui/",
"/swaggerapi/",
"/ui/",
"/version"
]
}
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0141e596414c gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube proxy -- 15 minutes ago Up 15 minutes nostalgic_nobel
10634ce798e9 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube schedule 16 minutes ago Up 16 minutes k8s_scheduler.b725e775_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_43562383
5618a39eb11d gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube apiserve 16 minutes ago Up 16 minutes k8s_apiserver.70750283_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_e5d145be
25f336102b26 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube controll 16 minutes ago Up 16 minutes k8s_controller-manager.aad1ee8f_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_fe538b9b
7f1391840920 gcr.io/google_containers/pause:0.8.0 "/pause" 17 minutes ago Up 17 minutes k8s_POD.e4cc795_k8s-master-127.0.0.1_default_9b44830745c166dfc6d027b0fc2df36d_26fd84fd
a11715435f45 gcr.io/google_containers/hyperkube:v0.21.2 "/hyperkube kubelet 17 minutes ago Up 17 minutes jovial_hodgkin
a882a1a4b917 gcr.io/google_containers/etcd:2.0.9 "/usr/local/bin/etcd 18 minutes ago Up 18 minutes adoring_hodgkin
docker 1.8.3 存在一些已知问题，尤其是 docker#17190. We had to workaround such issue through kubernetes#16052。但是这些变化并不是 Kubernetes 1.0 版本中特意挑选出来的。从您上面发布的输出中，我注意到没有暂停容器。您还可以 运行 docker ps -a 检查某些容器是否已死，然后将 docker logs <dead-container> 的输出复制并粘贴到此处？
我将提交一个问题以确保 Kubernetes 1.1 版本与 docker 1.8.3 一起正常工作。谢谢！