如何从 HA 集群和 etcd 集群中删除主节点

Question

我是k8s的新手，发现一个无法解决的问题

我正在构建一个 HA 主节点集群。我正在运行一些测试（删除一个节点并再次添加节点）。通过这个过程我注意到etcd集群没有更新集群列表。

下面的问题示例：

$ kubectl get pods -A
NAMESPACE                NAME                                                 READY   STATUS    RESTARTS   AGE
cri-o-metrics-exporter   cri-o-metrics-exporter-77c9cf9746-qlp4d              0/1     Pending   0          16h
haproxy-controller       haproxy-ingress-769d858699-b8r8q                     0/1     Pending   0          16h
haproxy-controller       ingress-default-backend-5fd4986454-kvbw8             0/1     Pending   0          16h
kube-system              calico-kube-controllers-574d679d8c-tkcjj             1/1     Running   3          16h
kube-system              calico-node-95t6l                                    1/1     Running   2          16h
kube-system              calico-node-m5txs                                    1/1     Running   2          16h
kube-system              coredns-7588b55795-gkfjq                             1/1     Running   2          16h
kube-system              coredns-7588b55795-lxpmj                             1/1     Running   2          16h
kube-system              etcd-masterNode1                                     1/1     Running   2          16h
kube-system              etcd-masterNode2                                     1/1     Running   2          16h
kube-system              kube-apiserver-masterNode1                           1/1     Running   3          16h
kube-system              kube-apiserver-masterNode2                           1/1     Running   3          16h
kube-system              kube-controller-manager-masterNode1                  1/1     Running   4          16h
kube-system              kube-controller-manager-masterNode2                  1/1     Running   4          16h
kube-system              kube-proxy-5q6xs                                     1/1     Running   2          16h
kube-system              kube-proxy-k8p6h                                     1/1     Running   2          16h
kube-system              kube-scheduler-masterNode1                           1/1     Running   3          16h
kube-system              kube-scheduler-masterNode2                           1/1     Running   6          16h
kube-system              metrics-server-575bd7f776-jtfsh                      0/1     Pending   0          16h
kubernetes-dashboard     dashboard-metrics-scraper-6f78bc588b-khjjr           1/1     Running   2          16h
kubernetes-dashboard     kubernetes-dashboard-978555c5b-9jsxb                 1/1     Running   2          16h
$ kubectl exec etcd-masterNode2 -n kube-system -it -- sh
sh-5.0# etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list -w table
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |            NAME            |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
| 4c209e5bc1ca9593 | started |         masterNode1        |     https://IP1:2380      |     https://IP1:2379      |      false |
| 676d4bfab319fa22 | started |         masterNode2        |     https://IP2:2380      |     https://IP2:2379      |      false |
| a9af4b00e33f87d4 | started |         masterNode3        |     https://IP3:2380      |     https://IP3:2379      |      false |
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
sh-5.0# exit
$ kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
masterNode1                  Ready    master   16h   v1.19.0
masterNode2                  Ready    master   16h   v1.19.0

我假设我正在从集群中正确删除节点。我正在执行的程序：

kubectl drain --ignore-daemonsets --delete-local-data
kubectl 删除
节点 kubeadm 重置
rm -f /etc/cni/net.d/* # 删除 CNI 配置
rm -rf /var/lib/kubelet # 删除 /var/lib/kubeler 目录
rm -rf /var/lib/etcd # 删除 /var/lib/etcd
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X && iptables -t filter -F && iptables -t filter -X # 删除 iptables
ipvsadm --clear
rm -rf /etc/kubernetes # 删除 /etc/kubernetes （在字符更改的情况下）

我是运行 kubernetes，版本 1.19.0 和 etcd etcd:3.4.9-1。

集群运行在裸机节点上。

这是一个错误还是我没有从 etcd 集群中正确删除节点？

Answer 1

感谢 Mariusz K. 我找到了问题的答案。如果其他人可能遇到同样的问题，我就是这样解决的。

首先查询 etcd 成员的集群 (HA)（代码示例）：

$ kubectl exec etcd-< nodeNameMasterNode > -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list
1863b58e85c8a808, started, nodeNameMaster1, https://IP1:2380, https://IP1:2379, false
676d4bfab319fa22, started, nodeNameMaster2, https://IP2:2380, https://IP2:2379, false
b0c50c50d563ed51, started, nodeNameMaster3, https://IP3:2380, https://IP3:2379, false

然后一旦你有了节点列表，你就可以删除任何你想要的成员。代码示例：

kubectl exec etcd-nodeNameMaster1 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member remove b0c50c50d563ed51
Member b0c50c50d563ed51 removed from cluster d1e1de99e3d19634

我希望能够从 etcd 集群中删除一个成员，而无需连接到 pod 和运行辅助命令。这样我通过exec.

向pod执行命令

如何从 HA 集群和 etcd 集群中删除主节点

How to remove a master node from a HA cluster and also from etcd cluster

kubernetes

etcd