AKS 中的 Cert-manager ClusterIssuer 问题

Question

我在 clusterissuer（证书管理器版本 1.7.1）中收到此错误：

“获取 CA 颁发者的密钥对时出错：解码证书 PEM 块时出错”

我将 ca.crt、tls.crt 和 tls.key 存储在 Azure 的 Key Vault 中。

kubectl describe clusterissuer ca-issuer

  Ca:
    Secret Name:  cert-manager-secret
Status:
  Conditions:
    Last Transition Time:  2022-02-25T11:40:49Z
    Message:               Error getting keypair for CA issuer: error decoding certificate PEM block
    Observed Generation:   1
    Reason:                ErrGetKeyPair
    Status:                False
    Type:                  Ready
Events:
  Type     Reason         Age                  From          Message
  ----     ------         ----                 ----          -------
  Warning  ErrGetKeyPair  3m1s (x17 over 58m)  cert-manager  Error getting keypair for CA issuer: error decoding certificate PEM block
  Warning  ErrInitIssuer  3m1s (x17 over 58m)  cert-manager  Error initializing issuer: error decoding certificate PEM block

kubectl 获取集群发行者

NAME        READY   AGE
ca-issuer   False   69m

这是 clusterissuer yaml 文件：

ca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: cert-manager-secret

这是用于检索 ca.crt、tls.crt 和 tls.key

的 KeyVault yaml 文件

keyvauls.yaml

apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akscacrt
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akscacrt # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: ca.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akstlscrt
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akstlscrt # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: tls.crt # key to store object value in kubernetes secret
---
apiVersion: spv.no/v2beta1
kind: AzureKeyVaultSecret
metadata:
  name: secret-akstlskey
  namespace: cert-manager
spec:
  vault:
    name: kv-xx # name of key vault
    object:
      name: akstlskey # name of the akv object
      type: secret # akv object type
  output: 
    secret: 
      name: cert-manager-secret # kubernetes secret name
      dataKey: tls.key # key to store object value in kubernetes secret
---

这些是使用的证书：

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: argocd-xx
  namespace: argocd
spec:
  secretName: argocd-xx
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  commonName: "argocd.xx"
  dnsNames:
    - "argocd.xx"
  privateKey:
    size: 4096
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sonarqube-xx
  namespace: sonarqube
spec:
  secretName: "sonarqube-xx"
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
  commonName: "sonarqube.xx"
  dnsNames:
    - "sonarqube.xx"
  privateKey:
    size: 4096

我可以看到我可以从密钥库中检索证书的秘密：

kubectl get secret -n cert-manager cert-manager-secret -o yaml

apiVersion: v1
data:
  ca.crt: XXX
  tls.crt: XXX
  tls.key: XXX

此外，另一件奇怪的事情是，我在 sonarqube/argocd 名称空间中获得了我之前部署的其他机密，但我的部署文件中不再存在这些机密。我无法删除它们，当我尝试删除它们时，它们会自动重新创建。看起来它们存储在某种缓存中。我还尝试删除命名空间 akv2k8s/cert-manager 并删除 cert-manager/akv2k8s 控制器并重新安装它们，但在重新安装和应用部署后出现同样的问题...

kubectl get secret -n sonarqube

NAME                                      TYPE                                  DATA   AGE
cert-manager-secret                       Opaque                                3      155m
default-token-c8b86                       kubernetes.io/service-account-token   3      2d1h
sonarqube-xx-7v7dh   Opaque                                1      107m
sql-db-secret                             Opaque                                2      170m

kubectl get secret -n argocd   
NAME                                   TYPE                                  DATA   AGE
argocd-xx-7b5kb   Opaque                                1      107m
cert-manager-secret-argo               Opaque                                3      157m
default-token-pjb4z                    kubernetes.io/service-account-token   3      3d15h

kubectl describe certificate sonarqube-xxx -n sonarqube

Status:
  Conditions:
    Last Transition Time:        2022-02-25T11:04:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2022-02-25T11:04:08Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  sonarqube-xxx-7v7dh
Events:                          <none>

有什么想法吗？

谢谢。

Answer 1

我只是上传证书信息就弄明白了 ca.crt。 tls.crt和tls.key纯文本，没有BASE64编码 Azure 中的 Key Vault 机密。

当 AKV2K8S 从 Key Vault 中检索机密并存储在 Kubernetes 中时，它会自动以 BASE64 编码。

此致，

AKS 中的 Cert-manager ClusterIssuer 问题

Issue with Cert-manager ClusterIssuer in AKS

azure

kubernetes

kubernetes-helm

azure-aks

cert-manager