使用 haproxy 部署 HA kubernetes master 时，kubeadm init 卡在健康检查中

Question

我正在用 kubeadm 部署 HA kubernetes master(stacked etcd) ，我跟着官网说明： https://kubernetes.io/docs/setup/independent/high-availability/
目前在我的集群中计划了四个节点：

一个 HAProxy 服务器节点用于主负载平衡。
三个 etcd 堆叠主节点。

我使用以下配置部署了 haproxy：

global
    daemon
    maxconn 256

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend haproxy_kube
    bind *:6443
    mode tcp
    option tcplog
    timeout client  10800s
    default_backend masters

backend masters
    mode tcp
    option tcplog
    balance leastconn
    timeout server  10800s
    server master01 <master01-ip>:6443 check

我的kubeadm-config.yaml是这样的：

apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
  name: "master01"
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
  certSANs:
  - "<haproxyserver-dns>"
controlPlaneEndpoint: "<haproxyserver-dns>:6443"
networking:
  serviceSubnet: "172.24.0.0/16"
  podSubnet: "172.16.0.0/16"

我的初始命令是：

kubeadm init --config=kubeadm-config.yaml -v 11

但是在我运行在 master01 上执行上面的命令后，它一直在记录以下信息：

I0122 11:43:44.039849   17489 manifests.go:113] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0122 11:43:44.041038   17489 local.go:57] [etcd] wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
I0122 11:43:44.041068   17489 waitcontrolplane.go:89] [wait-control-plane] Waiting for the API server to be healthy
I0122 11:43:44.042665   17489 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0122 11:43:44.044971   17489 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab" 'https://<haproxyserver-dns>:6443/healthz?timeout=32s'
I0122 11:43:44.120973   17489 round_trippers.go:438] GET https://<haproxyserver-dns>:6443/healthz?timeout=32s  in 75 milliseconds
I0122 11:43:44.120988   17489 round_trippers.go:444] Response Headers:
I0122 11:43:44.621201   17489 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab" 'https://<haproxyserver-dns>:6443/healthz?timeout=32s'
I0122 11:43:44.703556   17489 round_trippers.go:438] GET https://<haproxyserver-dns>:6443/healthz?timeout=32s  in 82 milliseconds
I0122 11:43:44.703577   17489 round_trippers.go:444] Response Headers:
I0122 11:43:45.121311   17489 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab" 'https://<haproxyserver-dns>:6443/healthz?timeout=32s'
I0122 11:43:45.200493   17489 round_trippers.go:438] GET https://<haproxyserver-dns>:6443/healthz?timeout=32s  in 79 milliseconds
I0122 11:43:45.200514   17489 round_trippers.go:444] Response Headers:
I0122 11:43:45.621338   17489 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab" 'https://<haproxyserver-dns>:6443/healthz?timeout=32s'
I0122 11:43:45.698633   17489 round_trippers.go:438] GET https://<haproxyserver-dns>:6443/healthz?timeout=32s  in 77 milliseconds
I0122 11:43:45.698652   17489 round_trippers.go:444] Response Headers:
I0122 11:43:46.121323   17489 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab" 'https://<haproxyserver-dns>:6443/healthz?timeout=32s'
I0122 11:43:46.199641   17489 round_trippers.go:438] GET https://<haproxyserver-dns>:6443/healthz?timeout=32s  in 78 milliseconds
I0122 11:43:46.199660   17489 round_trippers.go:444] Response Headers:

使用 Ctrl-C 退出循环后，我手动运行 curl 命令，但一切似乎都正常：

curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab" 'https://<haproxyserver-dns>:6443/healthz?timeout=32s'
* About to connect() to <haproxyserver-dns> port 6443 (#0)
*   Trying <haproxyserver-ip>...
* Connected to <haproxyserver-dns> (10.135.64.223) port 6443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*   subject: CN=kube-apiserver
*   start date: Jan 22 03:43:38 2019 GMT
*   expire date: Jan 22 03:43:38 2020 GMT
*   common name: kube-apiserver
*   issuer: CN=kubernetes
> GET /healthz?timeout=32s HTTP/1.1
> Host: <haproxyserver-dns>:6443
> Accept: application/json, */*
> User-Agent: kubeadm/v1.13.2 (linux/amd64) kubernetes/cff46ab
> 
< HTTP/1.1 200 OK
< Date: Tue, 22 Jan 2019 04:09:03 GMT
< Content-Length: 2
< Content-Type: text/plain; charset=utf-8
< 
* Connection #0 to host <haproxyserver-dns> left intact
ok

我不知道如何找出这个问题的本质原因，希望知道的人能给我一些建议。谢谢！

Answer 1

经过几天的查找和尝试，我又可以自己解决这个问题了。事实上，问题可能出现在一种非常罕见的情况下：

I set proxy on master node in both /etc/profile and docker.service.d, which made the request to haproxy don't work well.

我不知道是哪个设置导致了这个问题。但是在添加了无代理规则后，问题解决了，kubeadm 在 haproxy 负载均衡器之后成功初始化了一个 master。这是我的代理设置：

/etc/profile:

...
export http_proxy=http://<my-proxy-server-dns:port>/
export no_proxy=<my-k8s-master-loadbalance-server-dns>,<my-proxy-server-dns>,localhost

/etc/systemd/system/docker.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://<my-proxy-server-dns:port>/" "NO_PROXY<my-k8s-master-loadbalance-server-dns>,<my-proxy-server-dns>,localhost, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16"

使用 haproxy 部署 HA kubernetes master 时，kubeadm init 卡在健康检查中

kubeadm init stuck at health check when deploying HA kubernetes master with haproxy

kubernetes

kubeadm