Cert-Manager k8s,更新证书流程

Cert-Manager k8s, Renewing certificate process

我已经在 k8s 集群上安装了证书管理器:

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.5.3 --set installCRDs=true

我的objective是在同名微服务运行之间进行mtls通信-space.

为此,我创建了一个 ca 发行者 .i.e..

kubectl get issuer -n sandbox -o yaml
apiVersion: v1
items:
- apiVersion: cert-manager.io/v1
  kind: Issuer
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"cert-manager.io/v1","kind":"Issuer","metadata":{"annotations":{},"name":"ca-issuer","namespace":"sandbox"},"spec":{"ca":{"secretName":"tls-internal-ca"}}}
    creationTimestamp: "2021-09-16T17:24:58Z"
    generation: 1
    managedFields:
    - apiVersion: cert-manager.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:spec:
          .: {}
          f:ca:
            .: {}
            f:secretName: {}
      manager: HashiCorp
      operation: Update
      time: "2021-09-16T17:24:58Z"
    - apiVersion: cert-manager.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          .: {}
          f:conditions: {}
      manager: controller
      operation: Update
      time: "2021-09-16T17:24:58Z"
    name: ca-issuer
    namespace: sandbox
    resourceVersion: "3895820"
    selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/issuers/ca-issuer
    uid: 90f0c811-b78d-4346-bb57-68bf607ee468
  spec:
    ca:
      secretName: tls-internal-ca
  status:
    conditions:
      message: Signing CA verified
      observedGeneration: 1
      reason: KeyPairVerified
      status: "True"
      type: Ready
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

使用这个 ca 颁发者,我已经为我的两个微服务创建了证书,即

kubectl get certificate -n sandbox      
NAME                       READY         SECRET             Age
service1-certificate        True    service1-certificate     3d     
service2-certificate        True    service2-certificate     2d23h

配置为

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  annotations:
    meta.helm.sh/release-name: service1
    meta.helm.sh/release-namespace: sandbox
  creationTimestamp: "2021-09-17T10:20:21Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  managedFields:
  - apiVersion: cert-manager.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
        f:labels:
          .: {}
          f:app.kubernetes.io/managed-by: {}
      f:spec:
        .: {}
        f:commonName: {}
        f:dnsNames: {}
        f:duration: {}
        f:issuerRef:
          .: {}
          f:kind: {}
          f:name: {}
        f:renewBefore: {}
        f:secretName: {}
        f:subject:
          .: {}
          f:organizations: {}
        f:usages: {}
    manager: Go-http-client
    operation: Update
    time: "2021-09-17T10:20:21Z"
  - apiVersion: cert-manager.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:privateKey: {}
      f:status:
        .: {}
        f:conditions: {}
        f:notAfter: {}
        f:notBefore: {}
        f:renewalTime: {}
        f:revision: {}
    manager: controller
    operation: Update
    time: "2021-09-20T05:14:12Z"
  name: service1-certificate
  namespace: sandbox
  resourceVersion: "5177051"
  selfLink: /apis/cert-manager.io/v1/namespaces/sandbox/certificates/service1-certificate
  uid: 0cf1ea65-92a1-4b03-944e-b847de2c80d9
spec:
  commonName: example.com
  dnsNames:
  - service1
  duration: 24h0m0s
  issuerRef:
    kind: Issuer
    name: ca-issuer
  renewBefore: 12h0m0s
  secretName: service1-certificate
  subject:
    organizations:
    - myorg
  usages:
  - client auth
  - server auth
status:
  conditions:
  - lastTransitionTime: "2021-09-20T05:14:13Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2021-09-21T05:14:13Z"
  notBefore: "2021-09-20T05:14:13Z"
  renewalTime: "2021-09-20T17:14:13Z"
  revision: 5

现在,正如您在配置中看到的那样,我已配置为在 12 小时内更新它们。但是,通过此自定义证书资源创建的机密仍然存在两天(创建它的第一天)。我在想这个 tls 秘密将每天通过证书管理器更新).i.e.

kubectl get secrets service1-certificate service2-certificate -n sandbox -o wide
NAME                        TYPE                DATA   AGE
service1-certificate        kubernetes.io/tls   3      2d23h
service2-certificate        kubernetes.io/tls   3      3d1h

是不是我的理解有问题?在 certmangager pod 日志中,我确实看到了一些关于更新的错误。即

I0920 05:14:04.649158       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key”=“sandbox/service1-certificate" "message"="Renewing certificate as renewal was scheduled at 2021-09-19 08:24:13 +0000 UTC" "reason"="Renewing"
    
I0920 05:14:04.649235       1 conditions.go:201] Setting lastTransitionTime for Certificate “service1-certificate" condition "Issuing" to 2021-09-20 05:14:04.649227766 +0000 UTC m=+87949.327215532
    
I0920 05:14:04.652174       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="sandbox/service2 "message"="Renewing certificate as renewal was scheduled at 2021-09-19 10:20:22 +0000 UTC" "reason"="Renewing"
    
I0920 05:14:04.652231       1 conditions.go:201] Setting lastTransitionTime for Certificate “service2-certificate" condition "Issuing" to 2021-09-20 05:14:04.652224302 +0000 UTC m=+87949.330212052
    
I0920 05:14:04.671111       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671094596 +0000 UTC m=+87949.349082328
I0920 05:14:04.671344       1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:04.671332206 +0000 UTC m=+87949.349319948
    
    
I0920 05:14:12.703039       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
    
I0920 05:14:12.703896       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.7038803 +0000 UTC m=+87957.381868045
    
    
I0920 05:14:12.749502       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
    
I0920 05:14:12.750096       1 conditions.go:190] Found status change for Certificate “service1-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:12.750082572 +0000 UTC m=+87957.428070303
    
I0920 05:14:13.009032       1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="sandbox/service1-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service1-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.117843       1 controller.go:161] cert-manager/controller/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox/service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service2-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.119366       1 conditions.go:190] Found status change for Certificate “service2-certificate" condition "Ready": "True" -> "False"; setting lastTransitionTime to 2021-09-20 05:14:13.119351795 +0000 UTC m=+87957.797339520
    
I0920 05:14:13.122820       1 controller.go:161] cert-manager/controller/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key”=“sandbox\service2-certificate" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \”service-certificate\": the object has been modified; please apply your changes to the latest version and try again"
    
I0920 05:14:13.123907       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service2-certificate-t92qh" condition "Approved" to 2021-09-20 05:14:13.123896104 +0000 UTC m=+87957.801883833
I0920 05:14:13.248082       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Approved" to 2021-09-20 05:14:13.248071551 +0000 UTC m=+87957.926059296
I0920 05:14:13.253488       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “serivce2-certificate-t92qh" condition "Ready" to 2021-09-20 05:14:13.253474153 +0000 UTC m=+87957.931461871
I0920 05:14:13.388001       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest “service1-certificate-p9stz" condition "Ready" to 2021-09-20 05:14:13.387983783 +0000 UTC m=+87958.065971525
    

简答

根据您提供的证书的日志和详细信息,可以肯定地说它按预期工作

注意证书中的revision: 5,这意味着证书已经更新了4次。如果您现在尝试查看那里,这将是 6 或 7,因为证书每 12 小时更新一次。

日志

第一个真正令人困惑的是 cert-manager pod 中的 error messages。这主要是嘈杂的消息,它们本身并没有真正的帮助。

在这里查看 Github issue comment and here github issue 3667

如果确实需要日志,应通过在 cert-manager 部署中将 args 设置为 --v=5 来增加 verbosity level。要编辑部署 运行,请执行以下命令:

kubectl edit deploy cert-manager -n cert-manager

如何检查certificate/secret

更新证书时,secret's 和证书的年龄不会改变,但内容会被编辑,例如 secret 中的 resourceVersion 和证书中的 revision

以下是检查证书是否已更新的选项:

  1. 通过在更新前后在 yaml 中获取秘密来检查:

    kubectl get secret example-certificate -o yaml > secret-before
    

然后 运行 diff 他们之间。可以看到 tls.crtresourceVersion 都更新了。

  1. 查看证书 revisiondates 的状态 (我将持续时间设置为可能的最短 1h 和 renewBefore 55m,因此它每 5 分钟更新一次):

     $ kubectl get cert example-cert -o yaml
     notAfter: "2021-09-21T14:05:24Z"
     notBefore: "2021-09-21T13:05:24Z"
     renewalTime: "2021-09-21T13:10:24Z"
     revision: 7
    
  2. 检查 certificate/secret 部署的命名空间中的事件:

    $ kubectl get events
    117s        Normal   Issuing             certificate/example-cert                The certificate has been successfully issued
    117s        Normal   Reused              certificate/example-cert                Reusing private key stored in existing Secret resource "example-staging-certificate"
    6m57s       Normal   Issuing             certificate/example-cert                Renewing certificate as renewal was scheduled at 2021-09-21 13:00:24 +0000 UTC
    6m57s       Normal   Requested           certificate/example-cert                Created new CertificateRequest resource "example-cert-bs8g6"
    117s        Normal   Issuing             certificate/example-cert                Renewing certificate as renewal was scheduled at 2021-09-21 13:05:24 +0000 UTC
    117s        Normal   Requested           certificate/example-cert                Created new CertificateRequest resource "example-cert-7x8cf" UTC
    
  3. certificaterequests:

    $ kubectl get certificaterequests
    NAME                 APPROVED   DENIED   READY   ISSUER      REQUESTOR                                         AGE
    example-cert-2pxdd   True                True    ca-issuer   system:serviceaccount:cert-manager:cert-manager   14m
    example-cert-54zzc   True                True    ca-issuer      system:serviceaccount:cert-manager:cert-manager   4m29s
    example-cert-8vjcm   True                True    ca-issuer      system:serviceaccount:cert-manager:cert-manager   9m29s
    
  4. 检查 cert-manager pod 中的日志以查看四个阶段:

    I0921 12:45:24.000726       1 trigger_controller.go:181] cert-manager/controller/certificates-trigger "msg"="Certificate must be re-issued" "key"="default/example-cert" "message"="Renewing certificate as renewal was scheduled at 2021-09-21 12:45:24 +0000 UTC" "reason"="Renewing"
    I0921 12:45:24.000761       1 conditions.go:201] Setting lastTransitionTime for Certificate "example-cert" condition "Issuing" to 2021-09-21 12:45:24.000756621 +0000 UTC m=+72341.194879378
    I0921 12:45:24.120503       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Approved" to 2021-09-21 12:45:24.12049391 +0000 UTC m=+72341.314616684
    I0921 12:45:24.154092       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "example-cert-mxvbm" condition "Ready" to 2021-09-21 12:45:24.154081971 +0000 UTC m=+72341.348204734
    

备注

非常重要的一点是,并非所有 issuers 都支持 durationrenewBefore 标志。例如。 letsencrypt 仍然无法使用它并且有 90 个默认天数。

Refence.