将 eksctl ClusterConfig 与 vpc-cni 插件一起使用并传递 maxPodsPerNode 以启动模板的正确方法？
Correct way of using eksctl ClusterConfig with vpc-cni addon and pass maxPodsPerNode to launch template?
由于 m5.xlarge VM (57) 的 pod 限制，我一直在尝试使用 vpc-cni 插件创建 EKS 集群。创建后我可以看到它被传递给 launchtemplate 对象但是当做一个节点描述它仍然可以分配以前的（错误的？）数字
集群配置：
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: exchange-develop
region: us-east-1
version: '1.21'
managedNodeGroups:
- name: default
labels:
worker: default
instanceType: m5.xlarge
desiredCapacity: 2
minSize: 2
maxSize: 4
tags:
'k8s.io/cluster-autoscaler/enabled': 'true'
'k8s.io/cluster-autoscaler/exchange-develop': 'owned'
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::658464581062:policy/eks-csi-driver-policy
- arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
- arn:aws:iam::658464581062:policy/ALBIngressControllerIAMPolicy
- arn:aws:iam::658464581062:policy/ExternalDNSPlicy
- arn:aws:iam::658464581062:policy/eks-cluster-autoscaler
maxPodsPerNode: 110
availabilityZones: ['us-east-1c', 'us-east-1d']
iam:
withOIDC: true
vpc:
cidr: 10.10.0.0/16
#autoAllocateIPv6: true
# disable public access to endpoint and only allow private access
clusterEndpoints:
publicAccess: true
privateAccess: true
addons:
- name: vpc-cni
version: '1.10.1'
使用已编辑数据启动模板：
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=***
--
Content-Type: text/x-shellscript
Content-Type: charset="us-ascii"
#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh
KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=<>
API_SERVER_URL=<>
K8S_CLUSTER_DNS_IP=<>
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default,worker=default,eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d --max-pods=58' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false
节点描述：
Name: ip-10-10-19-34.ec2.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=exchange-develop
alpha.eksctl.io/nodegroup-name=default
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5.xlarge
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=default
eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd
eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
kubernetes.io/arch=amd64
kubernetes.io/hostname=<<
kubernetes.io/os=linux
node.kubernetes.io/instance-type=m5.xlarge
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1c
worker=default
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 02 Dec 2021 10:22:20 -0300
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:40 -0300 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.10.19.34
ExternalIP: <<
Hostname: <<
InternalDNS: <<
ExternalDNS: <<
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 4
ephemeral-storage: 83873772Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 15921236Ki
pods: 58
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 3920m
ephemeral-storage: 76224326324
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 14904404Ki
pods: 58
System Info:
Machine ID: ec28ac2717ec395cdf5b4e37f7672569
System UUID: ec28ac27-17ec-395c-df5b-4e37f7672569
Boot ID: 50b3d3d9-5dfa-40b6-99c8-20873632c7ca
Kernel Version: 5.4.156-83.273.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.7
Kubelet Version: v1.21.5-eks-bc4871b
Kube-Proxy Version: v1.21.5-eks-bc4871b
ProviderID: aws:///<<<
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-9z7pw 25m (0%) 0 (0%) 0 (0%) 0 (0%) 61m
kube-system kube-proxy-2slc8 100m (2%) 0 (0%) 0 (0%) 0 (0%) 61m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 125m (3%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
查看可分配 pods 58...
那么，使用 eksctl 通过 vpc-cni 创建集群并将 maxPodsPerNode 参数传递给 ec2 启动模板的正确方法是什么？
编辑：
我尝试过的其他事情：
使用 vpc-cni addon 2 managedNodeGroups 从头开始创建集群
有 maxPodsPerNode 和没有，两者都将取值 58 和
不是 110
用eksctl create nodegroup再添加一个nodegroup，还是58
使用 EKS AWS 添加另一个节点组 UI，仍然是 58
Eksctl 版本 0.75.0
Kubectl 版本 1.21.2
对于 managedNodeGroup，您需要指定 AMI ID：
aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id --region us-east-1 --query "Parameter.Value" --output text
managedNodeGroups:
- name: default
...
maxPodsPerNode: 110
ami: ami-00836a7940260f6dd
overrideBootstrapCommand: |
#!/bin/bash
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd'
由于 m5.xlarge VM (57) 的 pod 限制，我一直在尝试使用 vpc-cni 插件创建 EKS 集群。创建后我可以看到它被传递给 launchtemplate 对象但是当做一个节点描述它仍然可以分配以前的（错误的？）数字
集群配置：
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: exchange-develop
region: us-east-1
version: '1.21'
managedNodeGroups:
- name: default
labels:
worker: default
instanceType: m5.xlarge
desiredCapacity: 2
minSize: 2
maxSize: 4
tags:
'k8s.io/cluster-autoscaler/enabled': 'true'
'k8s.io/cluster-autoscaler/exchange-develop': 'owned'
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::658464581062:policy/eks-csi-driver-policy
- arn:aws:iam::aws:policy/ElasticLoadBalancingFullAccess
- arn:aws:iam::658464581062:policy/ALBIngressControllerIAMPolicy
- arn:aws:iam::658464581062:policy/ExternalDNSPlicy
- arn:aws:iam::658464581062:policy/eks-cluster-autoscaler
maxPodsPerNode: 110
availabilityZones: ['us-east-1c', 'us-east-1d']
iam:
withOIDC: true
vpc:
cidr: 10.10.0.0/16
#autoAllocateIPv6: true
# disable public access to endpoint and only allow private access
clusterEndpoints:
publicAccess: true
privateAccess: true
addons:
- name: vpc-cni
version: '1.10.1'
使用已编辑数据启动模板：
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=***
--
Content-Type: text/x-shellscript
Content-Type: charset="us-ascii"
#!/bin/sh
set -ex
sed -i -E "s/^USE_MAX_PODS=\"\$\{USE_MAX_PODS:-true}\"/USE_MAX_PODS=false/" /etc/eks/bootstrap.sh
KUBELET_CONFIG=/etc/kubernetes/kubelet/kubelet-config.json
echo "$(jq ".maxPods=110" $KUBELET_CONFIG)" > $KUBELET_CONFIG
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=<>
API_SERVER_URL=<>
K8S_CLUSTER_DNS_IP=<>
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=exchange-develop,alpha.eksctl.io/nodegroup-name=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default,worker=default,eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d --max-pods=58' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false
节点描述：
Name: ip-10-10-19-34.ec2.internal
Roles: <none>
Labels: alpha.eksctl.io/cluster-name=exchange-develop
alpha.eksctl.io/nodegroup-name=default
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m5.xlarge
beta.kubernetes.io/os=linux
eks.amazonaws.com/capacityType=ON_DEMAND
eks.amazonaws.com/nodegroup=default
eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd
eks.amazonaws.com/sourceLaunchTemplateId=lt-0037c1eab7037898d
eks.amazonaws.com/sourceLaunchTemplateVersion=1
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c
kubernetes.io/arch=amd64
kubernetes.io/hostname=<<
kubernetes.io/os=linux
node.kubernetes.io/instance-type=m5.xlarge
topology.kubernetes.io/region=us-east-1
topology.kubernetes.io/zone=us-east-1c
worker=default
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 02 Dec 2021 10:22:20 -0300
Taints: <none>
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:18 -0300 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 02 Dec 2021 11:18:31 -0300 Thu, 02 Dec 2021 10:22:40 -0300 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 10.10.19.34
ExternalIP: <<
Hostname: <<
InternalDNS: <<
ExternalDNS: <<
Capacity:
attachable-volumes-aws-ebs: 25
cpu: 4
ephemeral-storage: 83873772Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 15921236Ki
pods: 58
Allocatable:
attachable-volumes-aws-ebs: 25
cpu: 3920m
ephemeral-storage: 76224326324
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 14904404Ki
pods: 58
System Info:
Machine ID: ec28ac2717ec395cdf5b4e37f7672569
System UUID: ec28ac27-17ec-395c-df5b-4e37f7672569
Boot ID: 50b3d3d9-5dfa-40b6-99c8-20873632c7ca
Kernel Version: 5.4.156-83.273.amzn2.x86_64
OS Image: Amazon Linux 2
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.7
Kubelet Version: v1.21.5-eks-bc4871b
Kube-Proxy Version: v1.21.5-eks-bc4871b
ProviderID: aws:///<<<
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system aws-node-9z7pw 25m (0%) 0 (0%) 0 (0%) 0 (0%) 61m
kube-system kube-proxy-2slc8 100m (2%) 0 (0%) 0 (0%) 0 (0%) 61m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 125m (3%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
attachable-volumes-aws-ebs 0 0
Events: <none>
查看可分配 pods 58...
那么，使用 eksctl 通过 vpc-cni 创建集群并将 maxPodsPerNode 参数传递给 ec2 启动模板的正确方法是什么？
编辑：
我尝试过的其他事情：
使用 vpc-cni addon 2 managedNodeGroups 从头开始创建集群 有 maxPodsPerNode 和没有，两者都将取值 58 和 不是 110
用eksctl create nodegroup再添加一个nodegroup，还是58
使用 EKS AWS 添加另一个节点组 UI，仍然是 58
Eksctl 版本 0.75.0 Kubectl 版本 1.21.2
对于 managedNodeGroup，您需要指定 AMI ID：
aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id --region us-east-1 --query "Parameter.Value" --output text
managedNodeGroups:
- name: default
...
maxPodsPerNode: 110
ami: ami-00836a7940260f6dd
overrideBootstrapCommand: |
#!/bin/bash
/etc/eks/bootstrap.sh exchange-develop --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup=default,eks.amazonaws.com/nodegroup-image=ami-00836a7940260f6dd'