Terraform：使用 azurerm_kubernetes_cluster 创建 AKS 群集时指定网络安全组规则

Question

我正在使用 Terraform azurerm 提供程序版本 1.19 创建 AKS 集群。我想在创建集群时指定网络安全组规则，但我不知道如何引用创建的安全组，因为生成的安全组被赋予了一个随机数字的名称。

类似于：

aks-agentpool-33577837-nsg

有没有办法引用创建的 nsg 或至少输出名称中使用的随机数？

创建集群的配置：

resource "azurerm_resource_group" "k8s" {
  name     = "${var.resource_group_name}"
  location = "${var.location}"
}

resource "azurerm_kubernetes_cluster" "k8s" {
  name                = "${var.cluster_name}"
  location            = "${azurerm_resource_group.k8s.location}"
  resource_group_name = "${azurerm_resource_group.k8s.name}"
  dns_prefix          = "${var.dns_prefix}"
  kubernetes_version  = "${var.kubernetes_version}"

  linux_profile {
    admin_username = "azureuser"

    ssh_key {
      key_data = "${file("${var.ssh_public_key}")}"
    }
  }

  agent_pool_profile {
    name    = "default"
    count   = "${var.agent_count}"
    vm_size = "${var.vm_size}"
    os_type = "Linux"
  }

  service_principal {
    client_id     = "${var.client_id}"
    client_secret = "${var.client_secret}"
  }

  tags {
    source      = "terraform"
    environment = "${var.environment}" 
  }
}

这会生成一个安全组，我想向其添加额外的规则。这是我想添加的规则，以便可以检查 nginx-controller 的 liveness 探测器。

resource "azurerm_network_security_rule" "nginx_liveness_probe" {
  name                        = "nginx_liveness"
  priority                    = 100 
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "${var.nginx_liveness_probe_port}"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${azurerm_kubernetes_cluster.k8s.node_resource_group}"
  network_security_group_name = How do I reference the auto-generated nsg ?
  description = "Allow access to nginx liveness probe"
}

Answer 1

我假设您的 AKC 已添加到您使用 Terraform 配置的 azurerm_resource_group。如果是这样，您可以将具有任意数量 azurerm_network_security_rule 的自定义 azurerm_network_security_group 添加到该资源组，如 detailed here。

示例：

resource "azurerm_resource_group" "test" {
  name     = "acceptanceTestResourceGroup1"
  location = "West US"
}

resource "azurerm_network_security_group" "test" {
  name                = "acceptanceTestSecurityGroup1"
  location            = "${azurerm_resource_group.test.location}"
  resource_group_name = "${azurerm_resource_group.test.name}"
}

resource "azurerm_network_security_rule" "test" {
  name                        = "test123"
  priority                    = 100
  direction                   = "Outbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "*"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${azurerm_resource_group.test.name}"
  network_security_group_name = "${azurerm_network_security_group.test.name}"
}

不幸的是，网络安全组数据源需要 name 参数，通配符似乎不受支持，否则也可以选择。

Answer 2

这里有点晚了，但刚遇到这个问题。因此，对于仍在寻找解决方案的任何人，这就是我最终为获得 AKS NSG 名称所做的事情：

将此添加到配置 AKS 的 *.tf 文件中：

resource "azurerm_network_security_rule" "http" {
  name                        = "YOUR_NAME"
  priority                    = 102
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "80"
  destination_port_range      = "*"
  source_address_prefixes     = "${var.ips}"
  destination_address_prefix  = "${azurerm_public_ip.ingress.ip_address}"
  resource_group_name         = "${azurerm_kubernetes_cluster.test.node_resource_group}"
  network_security_group_name = "${data.external.aks_nsg_name.result.output}"

  depends_on = ["azurerm_resource_group.test"]
}

# get the NSG name
data "external" "aks_nsg_name" {
  program = [
    "bash",
    "${path.root}/scripts/aks_nsg_name.sh"
  ]

  depends_on = [azurerm_resource_group.test]
}

在您的项目中创建 aks_nsg_name.sh 并添加以下内容：

#!/bin/bash 
OUTPUT=$(az network nsg list --query [].name -o tsv | grep aks-agentpool | head -n 1)
jq -n --arg output "$OUTPUT" '{"output":$output}'

Answer 3

回答您问题的解决方案：

data "azurerm_resources" "example" {
  resource_group_name = azurerm_kubernetes_cluster.example.node_resource_group

  type = "Microsoft.Network/networkSecurityGroups"
}

output name_nsg {
    value = data.azurerm_resources.example.resources.0.name
}

resource "azurerm_network_security_rule" "example" {
  name                        = "example"
  priority                    = 100
  direction                   = "Outbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "*"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = azurerm_kubernetes_cluster.example.node_resource_group
  network_security_group_name = data.azurerm_resources.example.resources.0.name
}

..然后以同样的方式添加所有规则。

一般来说，最好使用 Azure Kubernetes Service 对 Kubernetes Services 的创建做出反应的自动化方式，而不是使用更多的 Terraform（尽管下面的 Kubernetes yaml 也可以与 Kubernetes Terraform provider):

A network security group filters traffic for VMs, such as the AKS nodes. As you create Services, such as a LoadBalancer, the Azure platform automatically configures any network security group rules that are needed. Don't manually configure network security group rules to filter traffic for pods in an AKS cluster. Define any required ports and forwarding as part of your Kubernetes Service manifests, and let the Azure platform create or update the appropriate rules. You can also use network policies, as discussed in the next section, to automatically apply traffic filter rules to pods.

在当前设置中应该可以通过简单地为您想要公开的那些端口创建一个 Kubernetes 服务。

例如，当我部署入口控制器时，Kubernetes 服务的创建会触发 IP address/Loadbalancer 及其 NSG 的创建：

apiVersion: v1
kind: Service
metadata:
  name: ingress-ingress-nginx-controller
  namespace: ingress
spec:
  loadBalancerSourceRanges:
  - 8.8.8.8/32
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

通过创建映射到所需 pod 端口并指定 loadBalancerSourceRanges 的 Kubernetes 服务（类型 LoadBalancer），可以为您的自定义目标指定类似的设置。

apiVersion: v1
kind: Service
metadata:
  name: mycustomservice
  namespace: myownnamespace
spec:
  loadBalancerSourceRanges:
  - 8.8.8.8/32 # your source IPs
  - 9.9.9.9/32
  ports:
  - name: myaccessport
    port: 777
    protocol: TCP
    targetPort: mydestinationport
  selector:
    app.kubernetes.io/name: myapp
  type: LoadBalancer

另见 the issue in the azurerm provider GitHub

Terraform：使用 azurerm_kubernetes_cluster 创建 AKS 群集时指定网络安全组规则

Terraform: Specify network security group rules when creating an AKS cluster using azurerm_kubernetes_cluster

azure

terraform

terraform-provider-azure

azure-aks