Kubernetes Cron-Job 松弛通知

Question

我有一个创建 postgres 备份作业的 cronjob。我想通过 webhook 将 cronjob 状态为失败或成功的通知发送到松弛通道。如何添加条件或指定 Job 的状态并发送到 slack？我想下面的 curl 请求也可以工作，但如果您发现任何错误，请警告。

kind: CronJob
metadata:
  name: standup
spec:
  schedule: "* 17 * * 1-5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: standup
            image: busybox
            resources:
              requests:
                cpu: 1m
                memory: 100Mi
            env:
              - args: /bin/sh
              - -c 
              - curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/TQPCENFHP/
      restartPolicy: OnFailure

~ semural$ kubectl logs $pods -n database
The following backups are available in specified backup path:
Added `s3` successfully.
[2020-04-13 14:24:46 UTC]      0B postgresql-cluster/


NAME                                SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
postgresql-postgresql-helm-backup   0 0 * * *   False     0        8h              18h

NAME                                           COMPLETIONS   DURATION   AGE
postgresql-postgresql-helm-backup-1586822400   1/1           37s        8h
postgresql-postgresql-helm-backup-list         1/1           2s         18h
postgresql-postgresql-helm-pgmon               1/1           49s        18h

Answer 1

我想我们可以创建一个简单的脚本来获取 cronjob 状态：

import json
import os
from kubernetes import client, config, utils
from kubernetes.client.rest import ApiException
from api.exceptions import BatchApiNamespaceNotExistedException

class Constants:
    BACKOFF_LIMIT = 1
    STATUS_RUNNING = "RUNNING"
    STATUS_SUCCEED = "SUCCEED"
    STATUS_FAILED = "FAILED"
    STATUS_NOT_FOUND = "NOT FOUND"

class KubernetesApi:
    def __init__(self):
        try:
            config.load_incluster_config()
        except:
            config.load_kube_config()
        self.configuration = client.Configuration()
        self.api_instance = client.BatchV1Api(client.ApiClient(self.configuration))
        self.api_instance_v1_beta = client.BatchV1beta1Api(client.ApiClient(self.configuration))

    def get_job_status(self, job):
        if job is not None:
            total_failed_pod = job.status.failed or 0
            total_succeeded_pod = job.status.succeeded or 0
            if total_failed_pod + total_succeeded_pod < Constants.BACKOFF_LIMIT:
                return Constants.STATUS_RUNNING
            elif total_succeeded_pod > 0:
                return Constants.STATUS_SUCCEED
            return Constants.STATUS_FAILED
        return Constants.STATUS_NOT_FOUND

    def get_cron_job_status(self, namespace):
        try:
            cron_job_list = self.api_instance_v1_beta.list_namespaced_cron_job(namespace=namespace,
                                                                          watch=False)
        except ApiException as e:
            raise BatchApiNamespaceNotExistedException("Exception when calling BatchV1Api->list_namespaced_cron_job: %s\n" % e)

        for cron_job in cron_job_list.items:
          if cron_job.status.active is not None:
            for active_cron_job in cron_job.status.active:
              job = self.api_instance.read_namespaced_job(namespace=namespace,
                                                       name=active_cron_job.name)
              if job_status == Constants.STATUS_FAILED:
                # Do whatever you want in there
                print(job_status)

因此，如果状态为失败，那么我们可以将日志发送到 slack。

Answer 2

我认为你已经有了一个良好的开端。假设你有 curl 命令作为脚本，将第一个参数作为要发布的消息，你可以执行以下操作：

kind: CronJob
metadata:
  name: standup
spec:
  schedule: "* 17 * * 1-5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: standup
            image: busybox
            resources:
              requests:
                cpu: 1m
                memory: 100Mi
            env:
              - args: /bin/sh
              - -c 
              - run-job.py || notify-cron-job "FAIL" && notify-cron-job "SUCCESS"

Kubernetes Cron-Job 松弛通知

Kubernetes Cron-Job Slack Notification

kubernetes

kubernetes-cronjob