Prometheus 从不同的微服务 (metrics_paths) 但相同的目标（IP 和端口）收集指标

Question

我想从应用程序 my_app 的不同微服务中收集指标。

我希望能够将它们分组，例如获取一个环境中所有微服务的请求总和，或比较不同环境中同一微服务的指标。

dev_env.com/api/a/actuator/prometheus
dev_env.com/api/b/actuator/prometheus
dev_env.com/api/c/actuator/prometheus
test1_env.com/api/a/actuator/prometheus
test1_env.com/api/b/actuator/prometheus
test1_env.com/api/c/actuator/prometheus

目前我计划为 prometheus 配置与微服务一样多的作业，如下所示：

- job_name: "dev_a"
  metrics_path: "/api/a/actuator/prometheus"
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "a"

- job_name: "dev_b"
  metrics_path: "/api/b/actuator/prometheus"
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "b"
...

有没有办法只用一份工作就可以解决这个问题？在这种情况下拥有一份或多项工作是否正确？现实中微服务真的很多，环境很少。

目标：指标的字段为 env、microservice、app。然后我将能够轻松地操纵它们。如果有一份工作的解决方案，那么我不需要 app 字段，因为 job 字段会替换它。现在我需要 app 来区分指标与 my_app 和其他指标，例如卡夫卡，服务器。

我在上看到了类似的问题，但这里的区别是我对所有微服务使用相同的 ip 和端口。此外，我没有找到为一项工作指定多个 metrics_paths 的简便方法。

Answer 1

我想你可以尝试创建动态 metrics_path:

- job_name: "foo"
  relabel_configs:
  # this relabeling puts "microservice" label value in place of "" in /api//actuator/prometheus
  - source_labels: [microservice]
    target_label: __metrics_path__
    regex: (.+)
    replacement: /api//actuator/prometheus
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "a"
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "b"

可以进一步移动并避免重复同一主机，但配置会更难理解。然而，真正的重新标记来了 kung-fu:

# The idea behind this is to iterate not hosts, but different paths.
# To do so, we're going to write "microservice" label values into the array of targets,
# while the real target is going to be set with the "host" label.
- job_name: wicked
  static_configs:
    - targets: [a,b,c]
      labels:
        host: localhost
        app: my_app
        env: dev
  relabel_configs:
  # First, we're going to save target as microservice label
  - source_labels: [__address__]
    target_label: microservice
  # Then construct the metrics_path, just as before
  - source_labels: [microservice]
    target_label: __metrics_path__
    regex: (.+)
    replacement: /api//actuator/prometheus
  # Now we need to set the real address of the server instead of microservice name
  - source_labels: [host]
    target_label: __address__
  # Finally, get rid of the host label
  - regex: ^host$
    action: labeldrop

结果是：

Prometheus 从不同的微服务 (metrics_paths) 但相同的目标（IP 和端口）收集指标

Prometheus collect metrics from different microservices (metrics_paths) but same target (ip and port)

configuration

microservices

prometheus