Kubernetes：Prometheus Count pod 实例数

Question

在 Kubernetes-client Java api 中，我可以使用这种方式获取给定应用的可用和已部署 pod 实例总数：

ApiClient defaultClient = Configuration.getDefaultApiClient();
AppsV1beta1Api apiInstance = new AppsV1beta1Api();
...
try {
    AppsV1beta1DeploymentList result = apiInstance.listDeploymentForAllNamespaces(_continue, fieldSelector, includeUninitialized, labelSelector, limit, pretty, resourceVersion, timeoutSeconds, watch);
    foreach(ExtensionsV1beta1Deployment extensionsDeployment : result.getItems() ) {
        Map<String, String> labels = extensionsDeployment.getMetadata().getLabels();
        String appName = labels.getOrDefault("app", "");
        ExtensionsV1beta1DeploymentStatus status = extensionsDeployment.getStatus();
        int availablePods = status.getAvailableReplicas();
        int deployedPods = status.getReplicas();
        if ( availablePods != deployedPods) {
             // Generate an alert
        }
    }
} catch (ApiException e) {
    System.err.println("Exception when calling AppsV1beta1Api#listDeploymentForAllNamespaces");
    e.printStackTrace();
}

在上面的示例中，我将 availablePods 与 deployedPods 进行比较，如果它们不匹配，我会生成警报。

我如何使用 Prometheus 使用警报规则复制此逻辑 and/or Alertmanager 配置，它检查给定应用程序或作业的可用 pod 实例数，如果没有t匹配指定数量的实例，是否会触发警报？

指定的阈值可以是总的deployedPods，也可以来自另一个配置文件或模板。

Answer 1

我不知道如何对所有命名空间执行此操作，但对于一个命名空间，它看起来像：

curl -k -s 'https://prometheus-k8s/api/v1/query?query=(sum(kube_deployment_spec_replicas%7Bnamespace%3D%22default%22%7D)%20without%20(deployment%2C%20instance%2C%20pod))%20-%20(sum(kube_deployment_status_replicas_available%7Bnamespace%3D%22default%22%7D)%20without%20(deployment%2C%20instance%2C%20pod))'

这是对默认命名空间的 curl 请求。

警报配置如下所示：

groups:
- name: example
  rules:

  # Alert for any instance that is unreachable for >5 minutes.
  - alert: availablePods!=deployedPods
    expr: (sum(kube_deployment_spec_replicas{namespace="$Name_of_namespace"}) without (deployment, instance, pod)) - (sum(kube_deployment_status_replicas_available{namespace="$Name_of_namespace"}) without (deployment, instance, pod)) != 0
    for: 15m
    labels:
      severity: page
    annotations:
      summary: "availablePods are not equal deployedPods"
      description: "In namespace $Name_of_namespace more than 15 minutes availablePods are not equal deployedPods. "

不要忘记将变量 $Name_of_namespace 更改为要检查的命名空间名称。

Kubernetes：Prometheus Count pod 实例数

Kubernetes: Prometheus Count number of pod instances

kubernetes

prometheus

prometheus-alertmanager