Collectd 和 write_kafka 插件

Question

我正在尝试设置 collectd 以使用 write_kafka 插件为 kafka 生成数据。它部分起作用，这意味着它创建主题但无法在其中提供数据。这是配置文件：

Hostname    "dummy"
FQDNLookup   false
Interval     60

LoadPlugin "logfile"
<Plugin "logfile">
  LogLevel "debug"
  File "/etc/collectd/collectd.log"
  Timestamp true
</Plugin>

LoadPlugin cpu
LoadPlugin df
LoadPlugin disk
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin netlink
LoadPlugin network
LoadPlugin sensors
LoadPlugin swap
LoadPlugin thermal

LoadPlugin "write_kafka"
<Plugin "write_kafka">
  Property "metadata.broker.list" "localhost:9092"
  <Topic "system">
    Format JSON
  </Topic>
</Plugin>

LoadPlugin csv
<Plugin "csv">
  DataDir "/etc/collectd/csv"
  StoreRates true
</Plugin>

一次运行，在kafka端我有一个system-0分区显示如下：

ls -lash system-0/
total 12K
4.0K drwxr-xr-x 2 root root 4.0K Nov 19 12:30 .
4.0K drwxr-x--- 4 root root 4.0K Nov 19 12:33 ..
   0 -rw-r--r-- 1 root root  10M Nov 19 12:30 00000000000000000000.index
   0 -rw-r--r-- 1 root root    0 Nov 19 09:47 00000000000000000000.log
   0 -rw-r--r-- 1 root root  10M Nov 19 12:30 00000000000000000000.timeindex
4.0K -rw-r--r-- 1 root root    8 Nov 19 12:30 leader-epoch-checkpoint

在哪里可以看到日志文件是空的。

这是日志：

[2019-11-19 10:46:00] plugin_load: plugin "cpu" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "df" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "disk" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "interface" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "load" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "memory" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "netlink" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "network" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "sensors" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "swap" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "thermal" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "cpu" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "df" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "disk" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "interface" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "load" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "memory" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "netlink" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "network" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "sensors" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "swap" successfully loaded.
[2019-11-19 10:46:00] plugin_load: plugin "thermal" successfully loaded.
[2019-11-19 10:46:00] Systemd detected, trying to signal readyness.
[2019-11-19 10:46:00] Initialization complete, entering read-loop.
[2019-11-19 10:46:00] write_kafka plugin: created KAFKA handle : rdkafka#producer-1
[2019-11-19 10:46:00] write_kafka plugin: handle created for topic : system
[2019-11-19 10:46:00] [thrd:2997bc280c89:9092/0]: 2997bc280c89:9092/0: Failed to resolve '2997bc280c89:9092': Name or service not known
[2019-11-19 10:46:00] [thrd:2997bc280c89:9092/0]: 2997bc280c89:9092/0: Failed to resolve '2997bc280c89:9092': Name or service not known

有什么线索吗？

Answer 1

Kafka 配置设置不正确。我们需要通过将 advertised.listeners 设置为 PLAINTEXT://HOSTNAME:9092 来宣传听众。

Collectd 和 write_kafka 插件

Collectd and write_kafka plugin

collectd

apache-kafka