如何将 Kafka Exporter 作为数据源添加到 Grafana？

Question

我正在尝试使用 Kafka 和 Kafka Exporter 与 Grafana 的简化示例。我有一个 docker-compose.yml 类似于以下内容：

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - 'ALLOW_ANONYMOUS_LOGIN=yes'
    networks:
      - app-tier
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  kafka-exporter:
    build: kafka-exporter
    ports:
      - "9308:9308"
    networks:
      - app-tier
    entrypoint: ["run.sh"]
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    networks:
      - app-tier

其中 run.sh 是一个包装脚本，用于在 Kafka 启动时不断重试运行导出器，请参阅 https://github.com/khpeek/kafka-exporter-example。问题是，当我登录到 Grafana 并尝试使用 URL http://kafka-exporter:9308/metrics 添加 Prometheus 数据源时，出现错误：

Error reading Prometheus: bad_response: readObjectStart: expect { or n, but found <, error found in #1 byte of ...|<html> |..., bigger context ...|<html> <head><title>Kafka Exporter</title>|...

下面是它在 UI 中的样子：

Grafana 似乎忽略了 /metrics 路径并尝试直接从 http://kafka-exporter:9308 抓取数据，这确实看起来像错误消息描述的：

相比之下，/metrics 端点包含实际指标：

> curl localhost:9308/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 20
...

为什么会出现此错误？ Grafana不应该接路径吗？

Answer 1

您无法将 Grafana Prometheus 数据源直接连接到导出器。您需要设置一个 Prometheus 服务器，从该服务器抓取 kafka exporter 指标，最后将 Grafana 连接到 Prometheus 服务器。

Answer 2

为了显示@usuario 的答案的实现，docker-compose.yml 需要 Prometheus 服务，

version: '2'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    environment:
      - 'ALLOW_ANONYMOUS_LOGIN=yes'
    networks:
      - app-tier
  kafka:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  kafka-exporter:
    build: kafka-exporter
    ports:
      - "9308:9308"
    networks:
      - app-tier
    entrypoint: ["run.sh"]
  cli:
    image: 'bitnami/kafka:latest'
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - app-tier
  prometheus:
    image: bitnami/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - "./prometheus/prometheus.yml:/opt/bitnami/prometheus/conf/prometheus.yml"
    networks:
      - app-tier
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    networks:
      - app-tier

其中 prometheus/prometheus.yml 配置文件静态配置为抓取 Kafka Exporter：

global:
  scrape_interval: 10s
  scrape_timeout: 10s
  evaluation_interval: 1m

scrape_configs:
  - job_name: kafka-exporter
    metrics_path: /metrics
    honor_labels: false
    honor_timestamps: true
    sample_limit: 0
    static_configs:
      - targets: ['kafka-exporter:9308']

现在可以添加 Prometheus 数据源，并且可以查看来自 Kafka Exporter 的指标，例如 kafka_consumergroup_lag：

如何将 Kafka Exporter 作为数据源添加到 Grafana？

How to add the Kafka Exporter as a data source to Grafana?

apache-kafka

prometheus

grafana