kafka 和 JMX 出口商
kafka and JMX-exporter
我无法使用 JMX 导出器公开 kafka 指标。你能看看我的步骤并在需要的地方纠正我吗?
我正在按照 here 的步骤使用 JMX 导出器启用 kafka。
以下是我遵循的分步说明
#get kafka
wget kafka_2.11-2.0.0
# Download Prometheus JMX exporter:
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml
#Edit Prometheus JMX exporter config file; and append following lines
echo “- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w* name: kafka_producer_” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo “— pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo “— pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_” >> /opt/kafka/prometheus/kafka-0-8-2.yml
#start zookeeper in terminal 0
/opt/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
#start kafka broker in terminal 1
KAFKA_HEAP_OPTS=’”-Xmx1000M -Xms1000M”’
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7071
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
#start kafka consumer in terminal 2
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7072
/opt/kafka/bin/kafka-console-consumer.sh — bootstrap-server 0.0.0.0:9092 — topic test — from-beginning
#start kafka producer in terminal 3
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7073
/opt/kafka/bin/kafka-console-producer.sh — broker-list 0.0.0.0:9092 — topic test
经过上述步骤后,zookeeper 和 kafka 运行 没问题。
我可以在生产者终端中输入一条消息,它会在消费者控制台上收到。但是,在 Prometheus 上看不到 kafka 指标。为了调试这个,我通过
检查了端口 7071/2/3
netstat -tlnp | grep 7071
netstat -tlnp | grep 7072
netstat -tlnp | grep 7073
导致空白响应;这意味着没有服务正在使用上述端口。我觉得 JMX 导出器没有正确启用。
你能帮我解决以上问题吗?
从你的问题来看,你将变量放在各自的行上,而博客将它们放在同一行...
例如这是启动 Kafka 服务器的方法
KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"' KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml' JMX_PORT=7081 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
否则,您需要导出变量,以便子流程像您所做的那样提取它们 in your previous question,这似乎可以很好地公开指标
export KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"'
export KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml'
export JMX_PORT=7081
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
注意:您链接到的博客未使用 JMX_PORT
,但这些端口不能与导出器相同。
我还建议至少下载一个高于 0.3 的版本 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/
并使用 Kafka 2.0 的配置 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
旁注:netstat -tlnp | grep 707
会立即向您展示所有这些
感谢 cricket-007 的帮助。
为了简单起见,我在此处列出了我遵循的步骤
wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
mkdir /opt/kafka
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml
cd kafka
export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7071:/opt/kafka/prometheus/kafka-2_0_0.yml"
export KAFKA_HEAP_OPTS="-Xmx1000M -Xms1000M"
mv ../kafka-2_0_0.yml prometheus/
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
netstat -tlnpu |grep 70
tcp6 0 0 :::7071 :::* LISTEN 209455/java
udp6 0 0 :::40705 :::*
curl -s localhost_or_IP:7071 | grep -i kafka
long list of metrics will be dumped on stdout -
我无法使用 JMX 导出器公开 kafka 指标。你能看看我的步骤并在需要的地方纠正我吗? 我正在按照 here 的步骤使用 JMX 导出器启用 kafka。
以下是我遵循的分步说明
#get kafka
wget kafka_2.11-2.0.0
# Download Prometheus JMX exporter:
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml
#Edit Prometheus JMX exporter config file; and append following lines
echo “- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w* name: kafka_producer_” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo “— pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo “— pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_” >> /opt/kafka/prometheus/kafka-0-8-2.yml
#start zookeeper in terminal 0
/opt/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties
#start kafka broker in terminal 1
KAFKA_HEAP_OPTS=’”-Xmx1000M -Xms1000M”’
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7071
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
#start kafka consumer in terminal 2
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7072
/opt/kafka/bin/kafka-console-consumer.sh — bootstrap-server 0.0.0.0:9092 — topic test — from-beginning
#start kafka producer in terminal 3
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml”
JMX_PORT=7073
/opt/kafka/bin/kafka-console-producer.sh — broker-list 0.0.0.0:9092 — topic test
经过上述步骤后,zookeeper 和 kafka 运行 没问题。 我可以在生产者终端中输入一条消息,它会在消费者控制台上收到。但是,在 Prometheus 上看不到 kafka 指标。为了调试这个,我通过
检查了端口 7071/2/3netstat -tlnp | grep 7071
netstat -tlnp | grep 7072
netstat -tlnp | grep 7073
导致空白响应;这意味着没有服务正在使用上述端口。我觉得 JMX 导出器没有正确启用。
你能帮我解决以上问题吗?
从你的问题来看,你将变量放在各自的行上,而博客将它们放在同一行...
例如这是启动 Kafka 服务器的方法
KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"' KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml' JMX_PORT=7081 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
否则,您需要导出变量,以便子流程像您所做的那样提取它们 in your previous question,这似乎可以很好地公开指标
export KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"'
export KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml'
export JMX_PORT=7081
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
注意:您链接到的博客未使用 JMX_PORT
,但这些端口不能与导出器相同。
我还建议至少下载一个高于 0.3 的版本 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/
并使用 Kafka 2.0 的配置 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml
旁注:netstat -tlnp | grep 707
会立即向您展示所有这些
感谢 cricket-007 的帮助。 为了简单起见,我在此处列出了我遵循的步骤
wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
mkdir /opt/kafka
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml
cd kafka
export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7071:/opt/kafka/prometheus/kafka-2_0_0.yml"
export KAFKA_HEAP_OPTS="-Xmx1000M -Xms1000M"
mv ../kafka-2_0_0.yml prometheus/
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
netstat -tlnpu |grep 70
tcp6 0 0 :::7071 :::* LISTEN 209455/java
udp6 0 0 :::40705 :::*
curl -s localhost_or_IP:7071 | grep -i kafka
long list of metrics will be dumped on stdout -