在 AKS 中使用管理身份的 Azure Key Vault
Azure Key Vault using manage identity in AKS
我正在 Azure AKS 中进行常规部署，我想使用 keyvault 来存储我的机密以访问数据库。
这是我的部署文件：
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
selector:
matchLabels:
app: sonarqube
replicas: 1
template:
metadata:
labels:
app: sonarqube
spec:
containers:
- name: sonarqube
image: sonarqube:8.9-developer
resources:
requests:
cpu: 500m
memory: 1024Mi
limits:
cpu: 2000m
memory: 4096Mi
volumeMounts:
- mountPath: "/mnt/secrets/"
name: secrets-store-inline
- mountPath: "/opt/sonarqube/data/"
name: sonar-data-new
- mountPath: "/opt/sonarqube/extensions/plugins/"
name: sonar-extensions-new2
env:
- name: "SONARQUBE_JDBC_USERNAME"
valueFrom:
secretKeyRef:
name: test-secret
key: username
- name: "SONARQUBE_JDBC_PASSWORD"
valueFrom:
secretKeyRef:
name: test-secret
key: password
- name: "SONARQUBE_JDBC_URL"
valueFrom:
configMapKeyRef:
name: sonar-config
key: url
ports:
- containerPort: 9000
protocol: TCP
volumes:
- name: sonar-data-new
persistentVolumeClaim:
claimName: sonar-data-new
- name: sonar-extensions-new2
persistentVolumeClaim:
claimName: sonar-extensions-new2
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kv-provider"
这是我的秘密仓库class:
kind: SecretProviderClass
metadata:
name: azure-kv-provider
spec:
provider: azure
secretObjects:
- data:
- key: username
objectName: username
- key: password
objectName: password
secretName: test-secret
type: Opaque
parameters:
usePodIdentity: "false"
useAssignedIdentity: "true"
userAssignedIdentityID: "zzzz-zzzz-zzzz-zzzz-zzzz"
keyvaultName: "dbkvtz"
cloudName: ""
objects: |
array:
- |
objectName: test
objectType: secret
objectAlias: username
objectVersion: ""
- |
objectName: test
objectType: secret
objectAlias: password
objectVersion: ""
resourceGroup: "myresourcegroup"
subscriptionId: "yyyy-yyyy-yyyy-yyy-yyyy"
tenantId: "xxxx-xxxx-xxxx-xxx-xxxx"
其中“zzzz-zzzz-zzzz-zzzz-zzzz”是创建的托管身份的客户端 ID。
在我创建“dbkvtz”的 Key Vault 中，我通过“访问策略”添加了我创建的托管身份。另一方面，在“管理身份”中，我无法在“Azure 角色分配”中添加任何角色——未找到所选订阅的角色分配。不知道有没有必要在里面加什么角色
AKS 群集是为系统分配的托管标识设置的。我想使用托管身份访问密钥保管库，因此我创建了一个客户端 ID 为“zzzz-zzzz-zzzz-zzzz-zzzz”的托管身份（其中“z”是 0-9a-z 的值）。 =14=]
我不太熟悉 AKS 中的 keyvault 集成，所以我不确定配置是否正确。
我收到这个错误：
kubectl describe pods:
Normal Scheduled 19m default-scheduler Successfully assigned default/sonarqube-6bdb9cfc85-npbfw to aks-agentpool-16966606-vmss000000
Warning FailedMount 5m43s (x5 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline sonar-data-new sonar-extensions-new2 default-token-t45tw]: timed out waiting for the condition
Warning FailedMount 3m27s kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[default-token-t45tw secrets-store-inline sonar-data-new sonar-extensions-new2]: timed out waiting for the condition
Warning FailedMount 71s (x2 over 10m) kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[sonar-data-new sonar-extensions-new2 default-token-t45tw secrets-store-inline]: timed out waiting for the condition
Warning FailedMount 37s (x17 over 19m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-6bdb9cfc85-npbfw, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set
logs az aks show -g RG -n SonarQubeCluster
{
"aadProfile": null,
"addonProfiles": {
"azurepolicy": {
"config": null,
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
}
},
"httpApplicationRouting": {
"config": null,
"enabled": false,
"identity": null
},
"omsagent": {
"config": {
"logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
},
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
}
}
},
"agentPoolProfiles": [
{
"availabilityZones": [
"1"
],
"count": 2,
"enableAutoScaling": false,
"enableEncryptionAtHost": null,
"enableFips": false,
"enableNodePublicIp": null,
"enableUltraSsd": null,
"gpuInstanceProfile": null,
"kubeletConfig": null,
"kubeletDiskType": "OS",
"linuxOsConfig": null,
"maxCount": null,
"maxPods": 110,
"minCount": null,
"mode": "System",
"name": "agentpool",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
"nodeLabels": {},
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.7",
"osDiskSizeGb": 128,
"osDiskType": "Managed",
"osSku": "Ubuntu",
"osType": "Linux",
"podSubnetId": null,
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"proximityPlacementGroupId": null,
"scaleDownMode": null,
"scaleSetEvictionPolicy": null,
"scaleSetPriority": null,
"spotMaxPrice": null,
"tags": null,
"type": "VirtualMachineScaleSets",
"upgradeSettings": null,
"vmSize": "Standard_DS2_v2"
}
],
"apiServerAccessProfile": {
"authorizedIpRanges": null,
"enablePrivateCluster": false,
"enablePrivateClusterPublicFqdn": null,
"privateDnsZone": null
},
"autoScalerProfile": null,
"autoUpgradeProfile": null,
"azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
"disableLocalAccounts": null,
"diskEncryptionSetId": null,
"dnsPrefix": "SonarQubeCluster-dns",
"enablePodSecurityPolicy": null,
"enableRbac": true,
"extendedLocation": null,
"fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
"fqdnSubdomain": null,
"httpProxyConfig": null,
"id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
"identity": {
"principalId": "yy",
"tenantId": "rr",
"type": "SystemAssigned",
"userAssignedIdentities": null
},
"identityProfile": {
"kubeletidentity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
}
},
"kubernetesVersion": "1.20.7",
"linuxProfile": null,
"location": "southcentralus",
"maxAgentPools": 100,
"name": "SonarQubeCluster",
"networkProfile": {
"dnsServiceIp": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"loadBalancerProfile": {
"allocatedOutboundPorts": null,
"effectiveOutboundIPs": [
{
"id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
"resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
}
],
"idleTimeoutInMinutes": null,
"managedOutboundIPs": {
"count": 1
},
"outboundIPs": null,
"outboundIpPrefixes": null
},
"loadBalancerSku": "Standard",
"natGatewayProfile": null,
"networkMode": null,
"networkPlugin": "kubenet",
"networkPolicy": null,
"outboundType": "loadBalancer",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
},
"nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
"podIdentityProfile": null,
"powerState": {
"code": "Running"
},
"privateFqdn": null,
"privateLinkResources": null,
"provisioningState": "Succeeded",
"resourceGroup": "RG",
"securityProfile": null,
"servicePrincipalProfile": {
"clientId": "msi"
},
"sku": {
"name": "Basic",
"tier": "Free"
},
"type": "Microsoft.ContainerService/ManagedClusters",
"windowsProfile": null
}
知道哪里出了问题吗？
提前谢谢你。
我想知道当我创建一个启用了“系统分配的托管标识”选项的新 AKS 集群时，是否会自动创建一个新的“托管标识”？
我问这个是因为我没有使用任何其他 "Managed Identity"，而是我手动创建的那个。
这些是遵循的步骤：
创建新的“托管身份”
在“托管身份” - “访问控制 (IAM)” 或 ” Azure 角色分配 我没有添加任何角色的权限，所以我将其保留为默认值。
创建“Key vault”并添加几个“Secrets”。
在“Key Vault” - “访问策略”中为“托管标识”添加新的访问策略" 为代理池创建了一个新的访问策略 "SonarQubeCluster-agentpool"
当我检查 “AKSclusterName”->“属性” -> 并单击 “MC_xx_AKSclusterName_southcentralus” 时，似乎我没有权限，因为我收到此消息“您无权访问此资源。”
如果它有助于理解一点问题，我从以下位置获取日志：
az aks show -g RG -n SonarQubeCluster
{
"aadProfile": null,
"addonProfiles": {
"azurepolicy": {
"config": null,
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
}
},
"httpApplicationRouting": {
"config": null,
"enabled": false,
"identity": null
},
"omsagent": {
"config": {
"logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
},
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
}
}
},
"agentPoolProfiles": [
{
"availabilityZones": [
"1"
],
"count": 2,
"enableAutoScaling": false,
"enableEncryptionAtHost": null,
"enableFips": false,
"enableNodePublicIp": null,
"enableUltraSsd": null,
"gpuInstanceProfile": null,
"kubeletConfig": null,
"kubeletDiskType": "OS",
"linuxOsConfig": null,
"maxCount": null,
"maxPods": 110,
"minCount": null,
"mode": "System",
"name": "agentpool",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
"nodeLabels": {},
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.7",
"osDiskSizeGb": 128,
"osDiskType": "Managed",
"osSku": "Ubuntu",
"osType": "Linux",
"podSubnetId": null,
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"proximityPlacementGroupId": null,
"scaleDownMode": null,
"scaleSetEvictionPolicy": null,
"scaleSetPriority": null,
"spotMaxPrice": null,
"tags": null,
"type": "VirtualMachineScaleSets",
"upgradeSettings": null,
"vmSize": "Standard_DS2_v2"
}
],
"apiServerAccessProfile": {
"authorizedIpRanges": null,
"enablePrivateCluster": false,
"enablePrivateClusterPublicFqdn": null,
"privateDnsZone": null
},
"autoScalerProfile": null,
"autoUpgradeProfile": null,
"azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
"disableLocalAccounts": null,
"diskEncryptionSetId": null,
"dnsPrefix": "SonarQubeCluster-dns",
"enablePodSecurityPolicy": null,
"enableRbac": true,
"extendedLocation": null,
"fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
"fqdnSubdomain": null,
"httpProxyConfig": null,
"id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
"identity": {
"principalId": "yy",
"tenantId": "rr",
"type": "SystemAssigned",
"userAssignedIdentities": null
},
"identityProfile": {
"kubeletidentity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
}
},
"kubernetesVersion": "1.20.7",
"linuxProfile": null,
"location": "southcentralus",
"maxAgentPools": 100,
"name": "SonarQubeCluster",
"networkProfile": {
"dnsServiceIp": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"loadBalancerProfile": {
"allocatedOutboundPorts": null,
"effectiveOutboundIPs": [
{
"id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
"resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
}
],
"idleTimeoutInMinutes": null,
"managedOutboundIPs": {
"count": 1
},
"outboundIPs": null,
"outboundIpPrefixes": null
},
"loadBalancerSku": "Standard",
"natGatewayProfile": null,
"networkMode": null,
"networkPlugin": "kubenet",
"networkPolicy": null,
"outboundType": "loadBalancer",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
},
"nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
"podIdentityProfile": null,
"powerState": {
"code": "Running"
},
"privateFqdn": null,
"privateLinkResources": null,
"provisioningState": "Succeeded",
"resourceGroup": "RG",
"securityProfile": null,
"servicePrincipalProfile": {
"clientId": "msi"
},
"sku": {
"name": "Basic",
"tier": "Free"
},
"type": "Microsoft.ContainerService/ManagedClusters",
"windowsProfile": null
}
谢谢！
SecretProviderClass 中的 userAssignedIdentityID 必须是用户分配的 Kubelet 托管标识 ID（NodePool 的托管标识），而不是为您的 AKS 创建的托管标识，因为卷将被访问通过节点上的 kubelet。
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: azure-kvname-user-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<Kubelet identity ID>"
keyvaultName: "kvname"
您还需要为这个 Kubelet 身份分配角色：
resource "azurerm_role_assignment" "akv_kubelet" {
scope = azurerm_key_vault.akv.id
role_definition_name = "Key Vault Secrets Officer"
principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
}
或
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID
可以找到文档here for user-assigned identity and here for system-assigned identity.
我正在 Azure AKS 中进行常规部署，我想使用 keyvault 来存储我的机密以访问数据库。
这是我的部署文件：
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
selector:
matchLabels:
app: sonarqube
replicas: 1
template:
metadata:
labels:
app: sonarqube
spec:
containers:
- name: sonarqube
image: sonarqube:8.9-developer
resources:
requests:
cpu: 500m
memory: 1024Mi
limits:
cpu: 2000m
memory: 4096Mi
volumeMounts:
- mountPath: "/mnt/secrets/"
name: secrets-store-inline
- mountPath: "/opt/sonarqube/data/"
name: sonar-data-new
- mountPath: "/opt/sonarqube/extensions/plugins/"
name: sonar-extensions-new2
env:
- name: "SONARQUBE_JDBC_USERNAME"
valueFrom:
secretKeyRef:
name: test-secret
key: username
- name: "SONARQUBE_JDBC_PASSWORD"
valueFrom:
secretKeyRef:
name: test-secret
key: password
- name: "SONARQUBE_JDBC_URL"
valueFrom:
configMapKeyRef:
name: sonar-config
key: url
ports:
- containerPort: 9000
protocol: TCP
volumes:
- name: sonar-data-new
persistentVolumeClaim:
claimName: sonar-data-new
- name: sonar-extensions-new2
persistentVolumeClaim:
claimName: sonar-extensions-new2
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kv-provider"
这是我的秘密仓库class:
kind: SecretProviderClass
metadata:
name: azure-kv-provider
spec:
provider: azure
secretObjects:
- data:
- key: username
objectName: username
- key: password
objectName: password
secretName: test-secret
type: Opaque
parameters:
usePodIdentity: "false"
useAssignedIdentity: "true"
userAssignedIdentityID: "zzzz-zzzz-zzzz-zzzz-zzzz"
keyvaultName: "dbkvtz"
cloudName: ""
objects: |
array:
- |
objectName: test
objectType: secret
objectAlias: username
objectVersion: ""
- |
objectName: test
objectType: secret
objectAlias: password
objectVersion: ""
resourceGroup: "myresourcegroup"
subscriptionId: "yyyy-yyyy-yyyy-yyy-yyyy"
tenantId: "xxxx-xxxx-xxxx-xxx-xxxx"
其中“zzzz-zzzz-zzzz-zzzz-zzzz”是创建的托管身份的客户端 ID。
在我创建“dbkvtz”的 Key Vault 中，我通过“访问策略”添加了我创建的托管身份。另一方面，在“管理身份”中，我无法在“Azure 角色分配”中添加任何角色——未找到所选订阅的角色分配。不知道有没有必要在里面加什么角色
AKS 群集是为系统分配的托管标识设置的。我想使用托管身份访问密钥保管库，因此我创建了一个客户端 ID 为“zzzz-zzzz-zzzz-zzzz-zzzz”的托管身份（其中“z”是 0-9a-z 的值）。 =14=]
我不太熟悉 AKS 中的 keyvault 集成，所以我不确定配置是否正确。
我收到这个错误：
kubectl describe pods:
Normal Scheduled 19m default-scheduler Successfully assigned default/sonarqube-6bdb9cfc85-npbfw to aks-agentpool-16966606-vmss000000
Warning FailedMount 5m43s (x5 over 16m) kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline sonar-data-new sonar-extensions-new2 default-token-t45tw]: timed out waiting for the condition
Warning FailedMount 3m27s kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[default-token-t45tw secrets-store-inline sonar-data-new sonar-extensions-new2]: timed out waiting for the condition
Warning FailedMount 71s (x2 over 10m) kubelet Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[sonar-data-new sonar-extensions-new2 default-token-t45tw secrets-store-inline]: timed out waiting for the condition
Warning FailedMount 37s (x17 over 19m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-6bdb9cfc85-npbfw, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set
logs az aks show -g RG -n SonarQubeCluster
{
"aadProfile": null,
"addonProfiles": {
"azurepolicy": {
"config": null,
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
}
},
"httpApplicationRouting": {
"config": null,
"enabled": false,
"identity": null
},
"omsagent": {
"config": {
"logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
},
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
}
}
},
"agentPoolProfiles": [
{
"availabilityZones": [
"1"
],
"count": 2,
"enableAutoScaling": false,
"enableEncryptionAtHost": null,
"enableFips": false,
"enableNodePublicIp": null,
"enableUltraSsd": null,
"gpuInstanceProfile": null,
"kubeletConfig": null,
"kubeletDiskType": "OS",
"linuxOsConfig": null,
"maxCount": null,
"maxPods": 110,
"minCount": null,
"mode": "System",
"name": "agentpool",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
"nodeLabels": {},
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.7",
"osDiskSizeGb": 128,
"osDiskType": "Managed",
"osSku": "Ubuntu",
"osType": "Linux",
"podSubnetId": null,
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"proximityPlacementGroupId": null,
"scaleDownMode": null,
"scaleSetEvictionPolicy": null,
"scaleSetPriority": null,
"spotMaxPrice": null,
"tags": null,
"type": "VirtualMachineScaleSets",
"upgradeSettings": null,
"vmSize": "Standard_DS2_v2"
}
],
"apiServerAccessProfile": {
"authorizedIpRanges": null,
"enablePrivateCluster": false,
"enablePrivateClusterPublicFqdn": null,
"privateDnsZone": null
},
"autoScalerProfile": null,
"autoUpgradeProfile": null,
"azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
"disableLocalAccounts": null,
"diskEncryptionSetId": null,
"dnsPrefix": "SonarQubeCluster-dns",
"enablePodSecurityPolicy": null,
"enableRbac": true,
"extendedLocation": null,
"fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
"fqdnSubdomain": null,
"httpProxyConfig": null,
"id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
"identity": {
"principalId": "yy",
"tenantId": "rr",
"type": "SystemAssigned",
"userAssignedIdentities": null
},
"identityProfile": {
"kubeletidentity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
}
},
"kubernetesVersion": "1.20.7",
"linuxProfile": null,
"location": "southcentralus",
"maxAgentPools": 100,
"name": "SonarQubeCluster",
"networkProfile": {
"dnsServiceIp": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"loadBalancerProfile": {
"allocatedOutboundPorts": null,
"effectiveOutboundIPs": [
{
"id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
"resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
}
],
"idleTimeoutInMinutes": null,
"managedOutboundIPs": {
"count": 1
},
"outboundIPs": null,
"outboundIpPrefixes": null
},
"loadBalancerSku": "Standard",
"natGatewayProfile": null,
"networkMode": null,
"networkPlugin": "kubenet",
"networkPolicy": null,
"outboundType": "loadBalancer",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
},
"nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
"podIdentityProfile": null,
"powerState": {
"code": "Running"
},
"privateFqdn": null,
"privateLinkResources": null,
"provisioningState": "Succeeded",
"resourceGroup": "RG",
"securityProfile": null,
"servicePrincipalProfile": {
"clientId": "msi"
},
"sku": {
"name": "Basic",
"tier": "Free"
},
"type": "Microsoft.ContainerService/ManagedClusters",
"windowsProfile": null
}
知道哪里出了问题吗？
提前谢谢你。
我想知道当我创建一个启用了“系统分配的托管标识”选项的新 AKS 集群时，是否会自动创建一个新的“托管标识”？
我问这个是因为我没有使用任何其他 "Managed Identity"，而是我手动创建的那个。
这些是遵循的步骤：
创建新的“托管身份”
在“托管身份” - “访问控制 (IAM)” 或 ” Azure 角色分配 我没有添加任何角色的权限，所以我将其保留为默认值。
创建“Key vault”并添加几个“Secrets”。
在“Key Vault” - “访问策略”中为“托管标识”添加新的访问策略" 为代理池创建了一个新的访问策略 "SonarQubeCluster-agentpool"
当我检查 “AKSclusterName”->“属性” -> 并单击 “MC_xx_AKSclusterName_southcentralus” 时，似乎我没有权限，因为我收到此消息“您无权访问此资源。”
如果它有助于理解一点问题，我从以下位置获取日志：
az aks show -g RG -n SonarQubeCluster
{
"aadProfile": null,
"addonProfiles": {
"azurepolicy": {
"config": null,
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
}
},
"httpApplicationRouting": {
"config": null,
"enabled": false,
"identity": null
},
"omsagent": {
"config": {
"logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
},
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
}
}
},
"agentPoolProfiles": [
{
"availabilityZones": [
"1"
],
"count": 2,
"enableAutoScaling": false,
"enableEncryptionAtHost": null,
"enableFips": false,
"enableNodePublicIp": null,
"enableUltraSsd": null,
"gpuInstanceProfile": null,
"kubeletConfig": null,
"kubeletDiskType": "OS",
"linuxOsConfig": null,
"maxCount": null,
"maxPods": 110,
"minCount": null,
"mode": "System",
"name": "agentpool",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
"nodeLabels": {},
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.7",
"osDiskSizeGb": 128,
"osDiskType": "Managed",
"osSku": "Ubuntu",
"osType": "Linux",
"podSubnetId": null,
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"proximityPlacementGroupId": null,
"scaleDownMode": null,
"scaleSetEvictionPolicy": null,
"scaleSetPriority": null,
"spotMaxPrice": null,
"tags": null,
"type": "VirtualMachineScaleSets",
"upgradeSettings": null,
"vmSize": "Standard_DS2_v2"
}
],
"apiServerAccessProfile": {
"authorizedIpRanges": null,
"enablePrivateCluster": false,
"enablePrivateClusterPublicFqdn": null,
"privateDnsZone": null
},
"autoScalerProfile": null,
"autoUpgradeProfile": null,
"azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
"disableLocalAccounts": null,
"diskEncryptionSetId": null,
"dnsPrefix": "SonarQubeCluster-dns",
"enablePodSecurityPolicy": null,
"enableRbac": true,
"extendedLocation": null,
"fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
"fqdnSubdomain": null,
"httpProxyConfig": null,
"id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
"identity": {
"principalId": "yy",
"tenantId": "rr",
"type": "SystemAssigned",
"userAssignedIdentities": null
},
"identityProfile": {
"kubeletidentity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
}
},
"kubernetesVersion": "1.20.7",
"linuxProfile": null,
"location": "southcentralus",
"maxAgentPools": 100,
"name": "SonarQubeCluster",
"networkProfile": {
"dnsServiceIp": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"loadBalancerProfile": {
"allocatedOutboundPorts": null,
"effectiveOutboundIPs": [
{
"id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
"resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
}
],
"idleTimeoutInMinutes": null,
"managedOutboundIPs": {
"count": 1
},
"outboundIPs": null,
"outboundIpPrefixes": null
},
"loadBalancerSku": "Standard",
"natGatewayProfile": null,
"networkMode": null,
"networkPlugin": "kubenet",
"networkPolicy": null,
"outboundType": "loadBalancer",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
},
"nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
"podIdentityProfile": null,
"powerState": {
"code": "Running"
},
"privateFqdn": null,
"privateLinkResources": null,
"provisioningState": "Succeeded",
"resourceGroup": "RG",
"securityProfile": null,
"servicePrincipalProfile": {
"clientId": "msi"
},
"sku": {
"name": "Basic",
"tier": "Free"
},
"type": "Microsoft.ContainerService/ManagedClusters",
"windowsProfile": null
}
谢谢！
SecretProviderClass 中的 userAssignedIdentityID 必须是用户分配的 Kubelet 托管标识 ID（NodePool 的托管标识），而不是为您的 AKS 创建的托管标识，因为卷将被访问通过节点上的 kubelet。
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: azure-kvname-user-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<Kubelet identity ID>"
keyvaultName: "kvname"
您还需要为这个 Kubelet 身份分配角色：
resource "azurerm_role_assignment" "akv_kubelet" {
scope = azurerm_key_vault.akv.id
role_definition_name = "Key Vault Secrets Officer"
principal_id = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
}
或
export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID
可以找到文档here for user-assigned identity and here for system-assigned identity.