Azure AKS 部署引发了 "InsufficientSubnetSize" 错误
Azure AKS deployment raised an "InsufficientSubnetSize" error
我正在尝试通过 ARM 模板部署 Azure AKS 实例。
我需要将 AKS 实例集成到现有的 Vnet 中。
我有一个用于 AKS 服务的专用子网。
但是，部署失败并出现以下错误：
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed.
Please list deployment operations for details. Please see
https://aka.ms/arm-debug for usage details.","details":
[{"code":"BadRequest","message":"{\r\n \"code\": \"InsufficientSubnetSize\",\r\n
\"message\": \"Pre-allocated IPs 93 exceeds IPs available in Subnet 11\",\r\n
\"target\": \"agentPoolProfile.count\"\r\n}"}]}
我正在为 Vnet 使用以下地址 space：XX.XX.XX.0/24 (XX.XX.XX.0 - XX.XX.XX.255 有 256 个地址。
我在这个 Vnet 中有一组专用子网，每个子网都是 /28 掩码（11+5 地址深度）：
XX.XX.XX.0/28
XX.XX.XX.16/28
XX.XX.XX.64/28
XX.XX.XX.128/28
XX.XX.XX.144/28
XX.XX.XX.160/28
XX.XX.XX.176/28
计划在AKS中使用子网XX.XX.XX.144/28。
当前AKS实例ARM模板如下：
"resources": [
{
"type": "Microsoft.ContainerService/managedClusters",
"apiVersion": "2019-04-01",
"name": "[parameters('resourceName')]",
"location": "[parameters('location')]",
"dependsOn": [],
"tags": {},
"properties": {
"kubernetesVersion": "[parameters('kubernetesVersion')]",
"enableRBAC": "[parameters('enableRBAC')]",
"dnsPrefix": "[parameters('dnsPrefix')]",
"agentPoolProfiles": [
{
"name": "agentpool",
"osDiskSizeGB": "[parameters('osDiskSizeGB')]",
"count": "3",
"vmSize": "[parameters('agentVMSize')]",
"osType": "[parameters('osType')]",
"storageProfile": "ManagedDisks",
"maxPods": "30",
"vnetSubnetID": "/subscriptions/XXXXX/resourceGroups/XXXX/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/akssubnet"
}
],
"servicePrincipalProfile": {
"ClientId": "[parameters('servicePrincipalClientId')]",
"Secret": "[parameters('servicePrincipalClientSecret')]"
},
"networkProfile": {
"networkPlugin": "azure",
"serviceCidr": "10.0.0.0/16",
"dnsServiceIP": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16"
},
"addonProfiles": {
"httpApplicationRouting": {
"enabled": "[parameters('enableHttpApplicationRouting')]"
},
"omsagent": {
"enabled": "[parameters('enableOmsAgent')]",
"config": {
"logAnalyticsWorkspaceResourceID": "[parameters('omsWorkspaceId')]"
}
}
}
}
},
"subscriptionId": "[split(parameters('omsWorkspaceId'),'/')[2]]",
"resourceGroup": "[split(parameters('omsWorkspaceId'),'/')[4]]"
}
]
根据以下文章设置网络配置文件参数：Microsoft.ContainerService managedClusters template reference
10.0.0.0/16 的 CIDR 属于专用范围，不会干扰我现有的 Vnet 范围。
我需要有关如何处理此部署错误的建议。
更新：
我已经尝试使用 Vnet/subnets 的值进行部署，但仍然失败：
Upd2:
根据 MS documentation "Minimum number of pods on the initial cluster creation using Azure CNI type is 30" which leads to the following number of subnet range in my case according to the formula：(number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure) = (3+1) + ((3+1)*30) = 124
因此，即使 pods 的数量在 ARM 模板中设置为 1，30 的乘数将始终存在。
Upd3:
但是，由于我无法扩展现有的子网范围，因此我设法使用以下配置部署了 AKS 实例：
"parameters": {
"SvcCidr": {
"type": "string",
"defaultValue": "10.0.0.0/16",
"metadata": {
"description": "Maximum number of pods that can run on a node."
}
},
"PodCidr": {
"type": "string",
"defaultValue": "10.244.0.0/16",
"metadata": {
"description": "Maximum number of pods that can run on a node."
}
},
"DnsSvcIP": {
"type": "string",
"defaultValue": "10.0.0.10",
"metadata": {
"description": "Maximum number of pods that can run on a node."
}
},
"DockerCidr": {
"type": "string",
"defaultValue": "",
"variables": {
"vnetSubnetId": "[resourceId(resourceGroup().name, 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('vnetSubnetName'))]",
"resources": [
{
"type": "Microsoft.ContainerService/managedClusters",
"agentPoolProfiles": [
{
"vnetSubnetID": "[variables('vnetSubnetId')]",
"networkProfile": {
"networkPlugin": "[parameters('NetPlugin')]",
"serviceCidr": "[parameters('SvcCidr')]",
"podCidr": "[parameters('PodCidr')]",
"DNSServiceIP": "[parameters('DnsSvcIP')]",
"dockerBridgeCidr": "[parameters('DockerCidr')]"
这导致我的子网范围 IP 地址仅提供给集群节点，而 pods 将使用私有 IP 地址范围。
摘自文档：
Subnets:
Must be large enough to accommodate the nodes, pods, and all
Kubernetes and Azure resources that might be provisioned in your
cluster. For example, if you deploy an internal Azure Load Balancer,
its front-end IPs are allocated from the cluster subnet, not public
IPs. The subnet size should also take into account upgrade operations
or future scaling needs. To calculate the minimum subnet size
including an additional node for upgrade operations: (number of nodes
+ 1) + ((number of nodes + 1) * maximum pods per node that you configure)
Example for a 50 node cluster: (51) + (51 * 30 (default)) = 1,581 (/21
or larger)
Example for a 50 node cluster that also includes provision to scale up
an additional 10 nodes: (61) + (61 * 30 (default)) = 1,891 (/21 or
larger)
If you don't specify a maximum number of pods per node when you create
your cluster, the maximum number of pods per node is set to 30. The
minimum number of IP addresses required is based on that value. If you
calculate your minimum IP address requirements on a different maximum
value, see how to configure the maximum number of pods per node to set
this value when you deploy your cluster.
这意味着对于您的情况，您需要 30*4 + 4 = 124 个 ip 地址才能正常工作，但请记住，如果您想要添加 4 个节点并升级它是行不通的。如果您想扩展到 5 个节点，那将行不通。另外，这么小的子网有什么意义呢？您无需为子网大小付费，因此将它们设置得足够大不是问题
意味着你需要/25，技术上。 128-4（由 azure 保留）= 124 ;)
阅读：https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster
对于您的问题，当您使用 azure 模块网络时，如其他答案中有关计算方法所示，您的子网可能只有一个节点。但实际上，您的子网的 IP 地址数量不足以容纳一个节点。因为已经有pods需要默认创建AKS集群时的IP地址，比如metric server等
这样你就可以使用网络结节kubelet了。在这个模块中，只是节点需要在子网中的 IP 地址。只需使用此网络模块，您就可以根据需要拥有 3 个节点，并使用仅具有 8 个 IP 地址的现有子网。有关详细信息，请参阅 Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)。
我正在尝试通过 ARM 模板部署 Azure AKS 实例。
我需要将 AKS 实例集成到现有的 Vnet 中。
我有一个用于 AKS 服务的专用子网。
但是，部署失败并出现以下错误：
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed.
Please list deployment operations for details. Please see
https://aka.ms/arm-debug for usage details.","details":
[{"code":"BadRequest","message":"{\r\n \"code\": \"InsufficientSubnetSize\",\r\n
\"message\": \"Pre-allocated IPs 93 exceeds IPs available in Subnet 11\",\r\n
\"target\": \"agentPoolProfile.count\"\r\n}"}]}
我正在为 Vnet 使用以下地址 space：XX.XX.XX.0/24 (XX.XX.XX.0 - XX.XX.XX.255 有 256 个地址。
我在这个 Vnet 中有一组专用子网，每个子网都是 /28 掩码（11+5 地址深度）：
XX.XX.XX.0/28
XX.XX.XX.16/28
XX.XX.XX.64/28
XX.XX.XX.128/28
XX.XX.XX.144/28
XX.XX.XX.160/28
XX.XX.XX.176/28
计划在AKS中使用子网XX.XX.XX.144/28。
当前AKS实例ARM模板如下：
"resources": [
{
"type": "Microsoft.ContainerService/managedClusters",
"apiVersion": "2019-04-01",
"name": "[parameters('resourceName')]",
"location": "[parameters('location')]",
"dependsOn": [],
"tags": {},
"properties": {
"kubernetesVersion": "[parameters('kubernetesVersion')]",
"enableRBAC": "[parameters('enableRBAC')]",
"dnsPrefix": "[parameters('dnsPrefix')]",
"agentPoolProfiles": [
{
"name": "agentpool",
"osDiskSizeGB": "[parameters('osDiskSizeGB')]",
"count": "3",
"vmSize": "[parameters('agentVMSize')]",
"osType": "[parameters('osType')]",
"storageProfile": "ManagedDisks",
"maxPods": "30",
"vnetSubnetID": "/subscriptions/XXXXX/resourceGroups/XXXX/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/akssubnet"
}
],
"servicePrincipalProfile": {
"ClientId": "[parameters('servicePrincipalClientId')]",
"Secret": "[parameters('servicePrincipalClientSecret')]"
},
"networkProfile": {
"networkPlugin": "azure",
"serviceCidr": "10.0.0.0/16",
"dnsServiceIP": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16"
},
"addonProfiles": {
"httpApplicationRouting": {
"enabled": "[parameters('enableHttpApplicationRouting')]"
},
"omsagent": {
"enabled": "[parameters('enableOmsAgent')]",
"config": {
"logAnalyticsWorkspaceResourceID": "[parameters('omsWorkspaceId')]"
}
}
}
}
},
"subscriptionId": "[split(parameters('omsWorkspaceId'),'/')[2]]",
"resourceGroup": "[split(parameters('omsWorkspaceId'),'/')[4]]"
}
]
根据以下文章设置网络配置文件参数：Microsoft.ContainerService managedClusters template reference
10.0.0.0/16 的 CIDR 属于专用范围，不会干扰我现有的 Vnet 范围。
我需要有关如何处理此部署错误的建议。
更新：
我已经尝试使用 Vnet/subnets 的值进行部署，但仍然失败：
Upd2:
根据 MS documentation "Minimum number of pods on the initial cluster creation using Azure CNI type is 30" which leads to the following number of subnet range in my case according to the formula：(number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure) = (3+1) + ((3+1)*30) = 124
因此，即使 pods 的数量在 ARM 模板中设置为 1，30 的乘数将始终存在。
Upd3:
但是，由于我无法扩展现有的子网范围，因此我设法使用以下配置部署了 AKS 实例：
"parameters": {
"SvcCidr": {
"type": "string",
"defaultValue": "10.0.0.0/16",
"metadata": {
"description": "Maximum number of pods that can run on a node."
}
},
"PodCidr": {
"type": "string",
"defaultValue": "10.244.0.0/16",
"metadata": {
"description": "Maximum number of pods that can run on a node."
}
},
"DnsSvcIP": {
"type": "string",
"defaultValue": "10.0.0.10",
"metadata": {
"description": "Maximum number of pods that can run on a node."
}
},
"DockerCidr": {
"type": "string",
"defaultValue": "",
"variables": {
"vnetSubnetId": "[resourceId(resourceGroup().name, 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('vnetSubnetName'))]",
"resources": [
{
"type": "Microsoft.ContainerService/managedClusters",
"agentPoolProfiles": [
{
"vnetSubnetID": "[variables('vnetSubnetId')]",
"networkProfile": {
"networkPlugin": "[parameters('NetPlugin')]",
"serviceCidr": "[parameters('SvcCidr')]",
"podCidr": "[parameters('PodCidr')]",
"DNSServiceIP": "[parameters('DnsSvcIP')]",
"dockerBridgeCidr": "[parameters('DockerCidr')]"
这导致我的子网范围 IP 地址仅提供给集群节点，而 pods 将使用私有 IP 地址范围。
摘自文档：
Subnets:
Must be large enough to accommodate the nodes, pods, and all Kubernetes and Azure resources that might be provisioned in your cluster. For example, if you deploy an internal Azure Load Balancer, its front-end IPs are allocated from the cluster subnet, not public IPs. The subnet size should also take into account upgrade operations or future scaling needs. To calculate the minimum subnet size including an additional node for upgrade operations: (number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure)
Example for a 50 node cluster: (51) + (51 * 30 (default)) = 1,581 (/21 or larger)
Example for a 50 node cluster that also includes provision to scale up an additional 10 nodes: (61) + (61 * 30 (default)) = 1,891 (/21 or larger)
If you don't specify a maximum number of pods per node when you create your cluster, the maximum number of pods per node is set to 30. The minimum number of IP addresses required is based on that value. If you calculate your minimum IP address requirements on a different maximum value, see how to configure the maximum number of pods per node to set this value when you deploy your cluster.
这意味着对于您的情况，您需要 30*4 + 4 = 124 个 ip 地址才能正常工作，但请记住，如果您想要添加 4 个节点并升级它是行不通的。如果您想扩展到 5 个节点，那将行不通。另外，这么小的子网有什么意义呢？您无需为子网大小付费，因此将它们设置得足够大不是问题
意味着你需要/25，技术上。 128-4（由 azure 保留）= 124 ;)
阅读：https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster
对于您的问题，当您使用 azure 模块网络时，如其他答案中有关计算方法所示，您的子网可能只有一个节点。但实际上，您的子网的 IP 地址数量不足以容纳一个节点。因为已经有pods需要默认创建AKS集群时的IP地址，比如metric server等
这样你就可以使用网络结节kubelet了。在这个模块中，只是节点需要在子网中的 IP 地址。只需使用此网络模块，您就可以根据需要拥有 3 个节点，并使用仅具有 8 个 IP 地址的现有子网。有关详细信息，请参阅 Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS)。