Azure Key Vault 与 AKS 集成适用于 nginx 教程 Pod，但不适用于实际项目部署

Question

根据标题，我的集成工作遵循 the documentation。

我可以部署 nginx.yaml，大约 70 秒后我可以打印出秘密：

kubectl exec -it nginx -- cat /mnt/secrets-store/secret1

现在我正在尝试将它应用到 PostgreSQL 部署进行测试，我从 Pod description 中得到以下信息：

  Warning  FailedMount             3s    kubelet                  MountVolume.SetUp failed for volume "secrets-store01-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod staging/postgres-deployment-staging-69965ff767-8hmww, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get keyvault client: failed to get key vault token: nmi response failed with status code: 404, err: <nil>

并且来自 nmi logs：

E0221 22:54:32.037357       1 server.go:234] failed to get identities, error: getting assigned identities for pod staging/postgres-deployment-staging-69965ff767-8hmww in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0221 22:54:32.037409       1 server.go:192] status (404) took 80003389208 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1

不知道为什么，因为我基本上是将设置从 nignx.yaml 复制到 postgres.yaml。他们在这里：

# nginx.yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx
  namespace: staging
  labels:
    aadpodidbinding: aks-akv-identity-binding-selector
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: /mnt/secrets-store
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: aks-akv-secret-provider

# postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment-staging
  namespace: staging
  labels:
    aadpodidbinding: aks-akv-identity-binding-selector
spec:
  replicas: 1
  selector:
    matchLabels:
      component: postgres
  template:
    metadata:
      labels:
        component: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:13-alpine
          ports:
            - containerPort: 5432
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: /mnt/secrets-store
            readOnly: true
          - name: postgres-storage-staging
            mountPath: /var/postgresql
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: aks-akv-secret-provider
        - name: postgres-storage-staging
          persistentVolumeClaim:
            claimName: postgres-storage-staging

---
apiVersion: v1
kind: Service
metadata:
  name: postgres-cluster-ip-service-staging
  namespace: staging
spec:
  type: ClusterIP
  selector:
    component: postgres
  ports:
    - port: 5432
      targetPort: 5432

这里有什么问题的建议？

Answer 1

我的疏忽...aadpodidbinding 应该在 template: per:

https://azure.github.io/aad-pod-identity/docs/best-practices/#deploymenthttpskubernetesiodocsconceptsworkloadscontrollersdeployment

生成的 YAML 应该是：

# postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment-production
  namespace: production
spec:
  replicas: 1
  selector:
    matchLabels:
      component: postgres
  template:
    metadata:
      labels:
        component: postgres
        aadpodidbinding: aks-akv-identity-binding-selector
    spec:
      containers:
        - name: postgres
          image: postgres:13-alpine
          ports:
            - containerPort: 5432
          env: 
            - name: POSTGRES_DB_FILE
              value: /mnt/secrets-store/DEV-PGDATABASE
            - name: POSTGRES_USER_FILE
              value: /mnt/secrets-store/DEV-PGUSER
            - name: POSTGRES_PASSWORD_FILE
              value: /mnt/secrets-store/DEV-PGPASSWORD
            - name: POSTGRES_INITDB_ARGS
              value: "-A md5"
            - name: PGDATA
              value: /var/postgresql/data
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: /mnt/secrets-store
            readOnly: true
          - name: postgres-storage-production
            mountPath: /var/postgresql
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: aks-akv-secret-provider
        - name: postgres-storage-production
          persistentVolumeClaim:
            claimName: postgres-storage-production
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-cluster-ip-service-production
  namespace: production
spec:
  type: ClusterIP
  selector:
    component: postgres
  ports:
    - port: 5432
      targetPort: 5432

Answer 2

在规范中添加模板将解决问题，在 deployment.yaml 文件

的模板标签部分使用标签“aadpodidbinding:”你的 azure pod 身份选择器“

示例部署文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        aadpodidbinding: azure-pod-identity-binding-selector
    spec:
      containers:
        - name: nginx
          image: nginx
          env:
          - name: SECRET
            valueFrom:
              secretKeyRef:
                name: test-secret
                key: key
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: dev-1spc

Azure Key Vault 与 AKS 集成适用于 nginx 教程 Pod，但不适用于实际项目部署

Azure Key Vault integration with AKS works for nginx tutorial Pod, but not actual project deployment

azure

kubernetes

azure-aks

azure-keyvault

aad-pod-identity