Kubernetes V1.11.0 HA 集群 CoreDNS PODS 未上线

Question

我正在尝试在 Centos 7 服务器上创建 Kubernetes V1.11.0 HA 集群。我在主机上禁用了 IPV6。

https://kubernetes.io/docs/setup/independent/high-availability/

基于这些步骤添加了所有三个 master，coreDNS POD 没有出现，它一直在 /var/log/messages 文件中记录以下错误。

kubectl get pods -n kube-system
NAME                                READY     STATUS              RESTARTS   AGE
coredns-78fcdf6894-8t7bt            0/1       ContainerCreating   0          53m
coredns-78fcdf6894-plmll            0/1       ContainerCreating   0          53m

错误信息：

Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531244   10041 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531286   10041 kuberuntime_sandbox.go:56] CreatePodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531298   10041 kuberuntime_manager.go:646] createPodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531358   10041 pod_workers.go:186] Error syncing pod 7caa3f29-89c3-11e8-aa0f-00505693ca30 ("coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)"), skipping: failed to "CreatePodSandbox" for "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)\" failed: rpc error: code = Unknown desc = [failed to set up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to set up pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to teardown pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: failed to get IP addresses for \"eth0\": <nil>]"
Jul 17 10:09:28 master03 kubelet: W0717 10:09:28.636173   10041 cni.go:243] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "88183d8a24dcafca5b04e3a6454e83aca1417a6934f50cd52b3e6fffbae4f272"
Jul 17 10:09:28 master03 kernel: XFS (dm-18): Mounting V4 Filesystem

如何修复此错误消息并引入 coreDNS PODs？

谢谢

Answer 1

Coreos Github 上有一个问题恰好描述了您的情况。解决问题的建议是：

squeed: You are using an old version of CNI, which does not work on systems with ipv6 disabled. Please re-enable ipv6 or upgrade CNI.

关于 CNI Github 上的相关问题，他还提到：

squeed: As I already explained, this has been fixed for 6 months. Please upgrade your CNI plugins.

并且jellonek评论道：

This is the question for your deployment software provider, which in your case is probably https://github.com/kubernetes/kubeadm

然而，如果您出于某种原因需要关闭 ipv6，您可以使用 addon KubeDNS instead of CoreDNS using kubeadm:

安装集群

kubeadm init --pod-network-cidr=192.168.0.0/16 --feature-gates=CoreDNS=false

或者您可以删除CoreDNS部署和服务，然后应用KubeDNS yaml，这应该是generated或者根据您的实际集群配置进行调整。

这是 deployAddons.sh 文件中负责部署 kube-dns 插件的函数：

function deploy_dns {
  echo "Deploying DNS on Kubernetes"
  cp "${KUBE_ROOT}/cluster/addons/dns/kube-dns/kube-dns.yaml.sed" kube-dns.yaml
  sed -i -e "s/\$DNS_DOMAIN/${DNS_DOMAIN}/g" kube-dns.yaml
  sed -i -e "s/\$DNS_SERVER_IP/${DNS_SERVER_IP}/g" kube-dns.yaml

  KUBEDNS=`eval "${KUBECTL} get services --namespace=kube-system | grep kube-dns | cat"`

  if [ ! "$KUBEDNS" ]; then
    # use kubectl to create kube-dns addon
    ${KUBECTL} --namespace=kube-system create -f kube-dns.yaml

    echo "Kube-dns addon is successfully deployed."
  else
    echo "Kube-dns addon is already deployed. Skipping."
  fi

  echo
}

Answer 2

我必须为我的 CoreDNS pod 启用 IPv6 才能运行。

这是我遵循的步骤：

https://www.thegeekdiary.com/how-to-enable-ipv6-on-centos-rhel-7/

编辑 /etc/default/grub 并在行中将内核参数 ipv6.disable 的值从 1 更改为 0：

# grub2-mkconfig -o /boot/grub2/grub.cfg

# shutdown -r now

Answer 3

我在没有启用 IPv6 的情况下解决了这个问题（在 https://www.facebook.com/groups/k8skr 的帮助下）

所以，原因是，Ubuntu 的默认 Kubernetes CNI 是 0.6.0。但问题已通过 Kubernetes CNI 0.7.0 解决所以你可以通过从下面的站点下载更新它，并将 bin 文件替换为 /opt/cni/bin/

https://github.com/containernetworking/plugins/releases/tag/v0.7.1

至少，它对我有用:)

Answer 4

我参考这些链接解决了这个问题

查看 kubernates 故障排除指南和您使用的网络插件

https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-or-kube-dns-is-stuck-in-the-pending-state

https://kubernetes.io/docs/concepts/cluster-administration/addons/

并安装以下内容并检查

kubectl get pods -n kube-system

Kubernetes V1.11.0 HA 集群 CoreDNS PODS 未上线

Kubernetes V1.11.0 HA cluster CoreDNS PODS not coming up

docker

kubernetes

kubeadm

coredns