AKS 的密钥保管库问题

Question

几周前，我发布了关于使用 Key Vault（使用用户分配的托管身份方法）的 Kubernetes 部署的类似问题。问题已解决，但在尝试从头开始实施时，有些事情对我来说没有意义。

基本上我收到关于安装量的错误：

Volumes:
  sonar-data-new:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-data-new
    ReadOnly:   false
  sonar-extensions-new2:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sonar-extensions-new2
    ReadOnly:   false
  secrets-store-inline:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=azure-kv-provider
  default-token-zwxzg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zwxzg
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age               From               Message
  ----     ------       ----              ----               -------
  Normal   Scheduled    12s               default-scheduler  Successfully assigned default/sonarqube-d44d498f8-46mpz to aks-agentpool-35716862-vmss000000
  Warning  FailedMount  3s (x5 over 11s)  kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-d44d498f8-46mpz, err: rpc error: code = Unknown desc = failed to mountobjects, error: failed to get objectType:secret, objectName:username, objectVersion:: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://SonarQubeHelm.vault.azure.net/secrets/username/?api-version=2016-10-01: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_request","error_description":"Identity not found"}

这是我的 secret-class.yml 文件（keyvault 的名称是正确的）。另外 xxx-xxxx-xxx-xxx-xxxxx4b5ec83 是 AKS 托管标识的 objectID (SonarQubeHelm-agentpool)

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: azure-kv-provider
spec:
  provider: azure
  secretObjects:
   - data:
      - key: username
        objectName: username
      - key: password
        objectName: password
     secretName: test-secret
     type: Opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: "xxx-xxxx-xxx-xxx-xxxxx4b5ec83"
    keyvaultName: "SonarQubeHelm"
    cloudName: ""
    objects:  |
      array:
        - |
          objectName: username
          objectType: secret
          objectAlias: username
          objectVersion: ""
        - |
          objectName: password
          objectType: secret
          objectAlias: password
          objectVersion: ""
    resourceGroup: "rg-LD-sandbox"
    subscriptionId: "xxxx"
    tenantId: "yyyy"

这是我的deployment.yml文件

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sonarqube
  name: sonarqube
spec:
  selector:
    matchLabels:
      app: sonarqube
  replicas: 1
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      containers:
        - name: sonarqube
          image: sonarqube:8.9-developer
          resources:
            requests:
              cpu: 500m
              memory: 1024Mi
            limits:
              cpu: 2000m
              memory: 4096Mi
          volumeMounts:
          - mountPath: "/mnt/"
            name: secrets-store-inline
          - mountPath: "/opt/sonarqube/data/"
            name: sonar-data-new
          - mountPath: "/opt/sonarqube/extensions/plugins/"
            name: sonar-extensions-new2
          env:
          - name: "SONARQUBE_JDBC_USERNAME"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: username
          - name: "SONARQUBE_JDBC_PASSWORD"
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: password
          - name: "SONARQUBE_JDBC_URL"
            valueFrom:
              configMapKeyRef:
                name: sonar-config
                key: url
          ports:
          - containerPort: 9000
            protocol: TCP
      volumes:
      - name: sonar-data-new
        persistentVolumeClaim:
          claimName: sonar-data-new
      - name: sonar-extensions-new2
        persistentVolumeClaim:
          claimName: sonar-extensions-new2
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
           secretProviderClass: "azure-kv-provider"

我为 AKS 托管身份分配了适当的权限以访问密钥库（xxx-xxxx-xxx-xxx-xxxxx4b5ec83 是 objectID[= AKS 托管标识的 68=] - SonarQubeHelm-agentpool)

xxxx@Azure:~/clouddrive/kubernetes/sonarqubekeyvault$ az role assignment list --assignee xxx-xxxx-xxx-xxx-xxxxx4b5ec83 --all [ { "canDelegate": null, "condition": null, "conditionVersion": null, "description": null, "id": "/subscriptions/xxxx-xxx-xxx-xxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-x-xx-xx-xx86584218f", "name": "xxxx-xx-x-x-xx86584218f", "principalId": "xxx-xxx-x-xxx-xx3a4b5ec83", "principalName": "xxxx-xxxx-xxx-xxx-xx79a3906b8", "principalType": "ServicePrincipal", "resourceGroup": "rg-LD-sandbox", "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxx-xxxx-xxx-xxxx-xxxfe8e74483", "roleDefinitionName": "Key Vault Administrator", "scope": "/subscriptions/xxxx-xxx-xxx-xxxx-xxx2e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm", "type": "Microsoft.Authorization/roleAssignments" }, { "canDelegate": null, "condition": null, "conditionVersion": null, "description": null, "id": "/subscriptions/xxxx-xxxx-xxxx-xxxx-xxxe22e8804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm/providers/Microsoft.Authorization/roleAssignments/xxxx-xxxx-xxxx-xxxx-xxx5137f480", "name": "xxxx-xxxx-xxxx-xxxx-xx5137f480", "principalId": "xxxx-xxxx-xxxx-xxxx-xx3a4b5ec83", "principalName": "xxxx-xxxx-xxxx-xxxx-xx79a3906b8", "principalType": "ServicePrincipal", "resourceGroup": "rg-LD-sandbox", "roleDefinitionId": "/subscriptions/xxxx-xxxx-xxxx-xxxx-0e1e22e8804e/providers/Microsoft.Authorization/roleDefinitions/xxxx-xxxx-xxxx-xxxx-xx2c155cd7", "roleDefinitionName": "Key Vault Secrets Officer", "scope": "/subscriptions/xxxx/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm", "type": "Microsoft.Authorization/roleAssignments" } ]

这是关于我的 Key Vault 的信息。

az keyvault show --name SonarQubeHelm

{ "id": "/subscriptions/xxxx-xxxxx-xxxx-xxxx-xxxxxxxx804e/resourceGroups/rg-LD-sandbox/providers/Microsoft.KeyVault/vaults/SonarQubeHelm", "location": "xxxxx", "name": "SonarQubeHelm", "properties": { "accessPolicies": [ { "applicationId": null, "objectId": "xxxx-xxx-xxxx-xxxx-xxxxa4b5ec83", "permissions": { "certificates": [ "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "ManageContacts", "ManageIssuers", "GetIssuers", "ListIssuers", "SetIssuers", "DeleteIssuers", "Purge" ], "keys": [ "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "Decrypt", "Encrypt", "UnwrapKey", "WrapKey", "Verify", "Sign", "Purge" ], "secrets": [ "Get", "List", "Set", "Delete", "Recover", "Backup", "Restore", "Purge" ], "storage": null }, "tenantId": "xxxx-xxxx-xxxx-xxxx-xxxxxdb8c610" }, { "applicationId": null, "objectId": "xxxx-xxxx-xxxx-xxxx-xxxx531f67f8", "permissions": { "certificates": [ "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "ManageContacts", "ManageIssuers", "GetIssuers", "ListIssuers", "SetIssuers", "DeleteIssuers", "Purge" ], "keys": [ "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "Decrypt", "Encrypt", "UnwrapKey", "WrapKey", "Verify", "Sign", "Purge" ], "secrets": [ "Get", "List", "Set", "Delete", "Recover", "Backup", "Restore", "Purge" ], "storage": null }, "tenantId": "xxxxx-xxxxxx-xxx-xxx8db8c610" }, { "applicationId": null, "objectId": "xxx-xxxx-xxxx-xxx-xxxx0df6af9", "permissions": { "certificates": [ "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore", "ManageContacts", "ManageIssuers", "GetIssuers", "ListIssuers", "SetIssuers", "DeleteIssuers" ], "keys": [ "Get", "List", "Update", "Create", "Import", "Delete", "Recover", "Backup", "Restore" ], "secrets": [ "Get", "List", "Set", "Delete", "Recover", "Backup", "Restore" ], "storage": null }, "tenantId": "xxx-xxx-xxx-xxx-xxx8db8c610" } ], "createMode": null, "enablePurgeProtection": null, "enableRbacAuthorization": false, "enableSoftDelete": true, "enabledForDeployment": false, "enabledForDiskEncryption": false, "enabledForTemplateDeployment": false, "hsmPoolResourceId": null, "networkAcls": null, "privateEndpointConnections": null, "provisioningState": "Succeeded", "sku": { "family": "A", "name": "Standard" }, "softDeleteRetentionInDays": 90, "tenantId": "xxx-xxx-xxx-xxx-xxx68db8c610", "vaultUri": "https://sonarqubehelm.vault.azure.net/" }, "resourceGroup": "rg-LD-sandbox", "systemData": null, "tags": {}, "type": "Microsoft.KeyVault/vaults" }

这是 CSI pod 运行目前：

NAME READY STATUS RESTARTS AGE csi-secrets-store-provider-azure-1632148185-hggl4 1/1 Running 0 5h4m ingress-nginx-controller-65c4f84996-99pkh 1/1 Running 0 5h49m secrets-store-csi-driver-xsx2r 3/3 Running 0 5h4m sonarqube-d44d498f8-46mpz 0/1 ContainerCreating 0 26m

我使用了这个 CSI 驱动程序

helm install csi-secrets-store-provider-azure/csi-secrets-store-provider-azure --set secrets-store-csi-driver.syncSecret.enabled=true --generate-name

为我遵循的 AKS 托管身份分配适当的权限（以防万一我也使用了 clientID，但它不起作用）。但是，从前面的命令来看，托管身份的权限似乎是正确的。

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv) export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv) az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

在访问策略中的密钥库“SonarQubeHelm”中，这些是添加了所有权限的应用程序： SonarQubeHelm-agentpool 和 SonarQubeHelm

Secrets（在 Key Vault 内）中的密钥值为 username 和 password。

一切都在同一个区域、资源组和命名空间（默认），我正在使用 1 个节点集群。

知道这个错误吗？

提前致谢！

Answer 1

经过一些测试，我所遵循的过程似乎是正确的。很可能，我在 AKS 托管标识的角色分配中使用 principalId 而不是 clientId。

其他面临类似问题的要点：

检查 AKS 自动创建的托管标识是什么。检查 clientId；例如，
```
az vmss identity show -g MC_rg-LX-sandbox_SonarQubeHelm_southcentralus -n 
aks-agentpool-xxxxx62-vmss -o yaml
```
Remember that MC_** is the resource group that AKS creates automatically to keep AKS resources. VMSS is the Virtual Machine Scale Set; you can get it under the same resource group.
检查它是否具有访问您创建的 Key Vault 的正确权限：例如，（其中 xxxx-xxxx-xxx-xxx-xx79a3906b8 是托管身份 clientId):
```
az role assignment list --assignee xxxx-xxxx-xxx-xxx-xx79a3906b8 --all
```
它应该有：
```
"roleDefinitionName": "Key Vault Administrator"
```

如果没有正确的权限，分配它们：

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.clientId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

AKS 中托管标识的角色分配适用于 clientId。

AKS 的密钥保管库问题

key vault issue with AKS

azure

kubernetes

azure-keyvault

kubernetes-helm

azure-aks