使用 SecretProviderClass 的秘密创建不能按方面工作

Question

编辑：这是一个配置错误，我设置了错误的 kv 名称：/

如标题所述，我在使用 SecretProviderClass 创建秘密时遇到问题。

我已经在 azure 上创建了我的 aks 和我的 kv（并填充了它）。然后我将使用 user-assigned 托管身份

继续执行 those 步骤

但没有 secret 资源被创建并且 pods 因安装失败而卡在创建过程中。

这些是我遵循的步骤

az extension add --name aks-preview
az extension update --name aks-preview
az aks enable-addons --addons azure-keyvault-secrets-provider -g $RESOURCE_GROUP -n $AKS_CLUSTER 
az aks update -g $RESOURCE_GROUP -n $AKS_CLUSTER --enable-managed-identity --disable-secret-rotation
$AKS_ID = (az aks show -g $RESOURCE_GROUP -n $AKS_CLUSTER --query identityProfile.kubeletidentity.clientId -o tsv)
az keyvault set-policy -n $AZUREKEYVAULT --secret-permissions get --spn $AKS_ID

我正在使用的 SecretProviderClass 清单

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  secretObjects:
  - secretName: akvsecrets
    type: Opaque
    data:
    - objectName: AzureSignalRConnectionString 
      key: AzureSignalRConnectionString
    - objectName: BlobStorageConnectionString 
      key: BlobStorageConnectionString
    - objectName: SqlRegistryConnectionString 
      key: SqlRegistryConnectionString
    - objectName: TokenSymmetricKey 
      key: TokenSymmetricKey
  parameters:
    useVMManagedIdentity: "true"
    userAssignedIdentityID: XXX # VMSS UserAssignedIdentity
    keyvaultName: "sampleaks001" # the name of the KeyVault
    objects:  |
      array:
        - |
          objectName: AzureSignalRConnectionString
          objectType: secret
        - |
          objectName: BlobStorageConnectionString
          objectType: secret
        - |
          objectName: SqlRegistryConnectionString
          objectType: secret
        - |
          objectName: TokenSymmetricKey
          objectType: secret
    resourceGroup: sample # [REQUIRED for version < 0.0.4] the resource group of the KeyVault
    subscriptionId: XXXX # [REQUIRED for version < 0.0.4] the subscription ID of the KeyVault
    tenantId: XXX # the tenant ID of the KeyVault

和部署清单

apiVersion: apps/v1
kind: Deployment
metadata:
  name: trm-api-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: trm-api-test
  template:
    metadata:
      labels:
        app: trm-api-test
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
      - name: trm-api-test
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 80
        env:
        - name: AzureSignalRConnectionString
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: AzureSignalRConnectionString
        - name: TokenSymmetricKey
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: TokenSymmetricKey
        - name: BlobStorageConnectionString
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: BlobStorageConnectionString
        - name: SqlRegistryConnectionString
          valueFrom:
            secretKeyRef:
              name: akvsecrets
              key: SqlRegistryConnectionString
        volumeMounts:
        - name: secrets-store-inline
          mountPath: "/mnt/secrets-store"
          readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-kvname"
---
apiVersion: v1
kind: Service
metadata:
  name: trm-api-service-test
  namespace: default
spec:
  type: ClusterIP
  selector:
    app: trm-api-test
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP

我确定我遗漏了什么，但不明白是什么。提前致谢！

Answer 1

您使用的是 clientId，但它应该是 kubelet 身份中的 objectId：

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

这是我正在使用的有效 SecretProviderClass（根据您的配置进行了调整）：

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
spec:
  provider: azure
  secretObjects:
  - data:
    - objectName: AzureSignalRConnectionString 
      key: AzureSignalRConnectionString
    - objectName: BlobStorageConnectionString 
      key: BlobStorageConnectionString
    - objectName: SqlRegistryConnectionString 
      key: SqlRegistryConnectionString
    - objectName: TokenSymmetricKey 
      key: TokenSymmetricKey
    secretName: akvsecrets
    type: Opaque
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"
    userAssignedIdentityID: XXX # Kubelet Client Id ( Nodepool Managed Idendity )
    keyvaultName: "sampleaks001" # the name of the KeyVault
    tenantId: XXX # the tenant ID of the KeyVault  
    objects:  |
      array:
        - |
          objectName: AzureSignalRConnectionString
          objectAlias: AzureSignalRConnectionString
          objectType: secret
        - |
          objectName: BlobStorageConnectionString
          objectAlias: BlobStorageConnectionString
          objectType: secret
        - |
          objectName: SqlRegistryConnectionString
          objectAlias: SqlRegistryConnectionString
          objectType: secret
        - |
          objectName: TokenSymmetricKey
          objectAlias: TokenSymmetricKey
          objectType: secret

您还可以查看文档 here，因为您会在 Azure 文档中找到更好的示例。

使用 SecretProviderClass 的秘密创建不能按方面工作

Secret creation with SecretProviderClass not working as aspected

kubernetes

azure-aks

kubernetes-secrets