K8s 如何使用法兰绒每个节点使用超过 254 pods 个？

Question

我试图在我的集群中为每个节点使用 500+ pods。我已经将 maxPods 设置为 1k，从节点信息我可以看到这是成功的。

但是，flannel 不允许我使用超过 254，因为它为 pods 使用 24 个子网，范围为 x.1 到 x.255。

有办法实现吗？我尝试将 pod-network-cidr 增加到 8，并且还像这样配置了 flannel net-conf，

{                                                                                                                                                                   
  "Network": "10.0.0.0/8",                                                                                                                                         
  "SubnetLen": 16,                                                                                                                                                   
  "SubnetMin": "10.10.0.0",                                                                                                                                         
  "SubnetMax": "10.99.0.0",                                                                                                                                         
  "Backend": {                                                                                                                                                      
    "Type": "host-gw"                                                                                                                                               
  }                                                                                                                                                                 
} 

不过好像对我的集群没有影响。

这里是pending的事件输出pods

rpc error: code = Unknown desc = failed to set up sandbox
 container "f5721f2419ad98fc8f26ad42f3f142a5733db38fefacc6d6f75cc38bd890233a" network for pod "test-deployment-f69647ffc-2bgb6": networkPlugin cni failed to set up
pod "test-deployment-f69647ffc-2bgb6_default" network: failed to allocate for range 0: no IP addresses available in range set: 10.0.0.1-10.0.0.254                 
  Warning  FailedCreatePodSandBox  8m18s (x80 over 30m)   kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7045913eb55f3fbae10fe42de5fe29de73848e0cf005e96fa61286cdb127bc7d" network for pod "test-deployment-f69647ffc-2bgb6": net
workPlugin cni failed to set up pod "test-deployment-f69647ffc-2bgb6_default" network: failed to allocate for range 0: no IP addresses available in range set: 10.0.0.1-10.0.0.254 

Answer 1

通过更改 kube-controller-manager 的 node-cidr-mask-size 解决。

K8s:

将 --node-cidr-mask-size 16 传递给 kube-controller-manager 并将 --pod-network-cidr=10.0.0.0/8 传递给 kubeadm。像下面的示例一样编辑 kube-flannel.yml、net-conf.json 数组。

K3s:

export INSTALL_K3S_EXEC="--flannel-conf "<path-to-your-flannel-configuration>" --cluster-cidr "10.0.0.0/8" --kube-controller-manager-arg "node-cidr-mask-size=16"""

• 像下面的例子一样编辑你的 flannel 配置文件，然后照常安装 k3s。

法兰绒配置示例：

{
    "Network": "10.0.0.0/8",
    "Backend": {
        "Type": "vxlan"
    }
}