logstash kafka 输入不工作

Question

我正在尝试从 Kafka 获取数据并将其推送到 ElasticSearch。

这是我使用的 logstash 配置：

input {
kafka {
zk_connect => "localhost:2181"
topic_id => "beats"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "elasticse"
}
}

谁能帮忙配置logstash？如果我运行这是无效的配置错误。

D:\logstash-5.0.0\bin>logstash -f log-uf.conf
Sending Logstash logs to D:\logstash-5.0.0\logs\logstash-plain.txt which is now
configured via log4j2.properties.
[2016-11-11T16:31:32,429][ERROR][logstash.inputs.kafka    ] Unknown setting 'zk_
connect' for kafka
[2016-11-11T16:31:32,438][ERROR][logstash.inputs.kafka    ] Unknown setting 'top
ic_id' for kafka
[2016-11-11T16:31:32,452][ERROR][logstash.agent           ] fetched an invalid c
onfig {:config=>"input {\n   kafka {\n   zk_connect => \"localhost:2181\"\n   to
pic_id => \"beats\"\n   consumer_threads => 16\n  }\n}\noutput {\nelasticsearch
{\nhosts => [\"localhost:9200\"]\nindex => \"elasticse\"\n}\n}\n", :reason=>"Som
ething is wrong with your configuration."}

有人可以帮忙吗？

Answer 1

您是运行 Logstash 5，配置为 Logstash 2.4。

zk_connect（Zookeeper 主机）在 5.0

中被 bootstrap_servers（Kafka 代理）和 topic_id 替换为 topics

试试这个配置：

input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topics => ["beats"]
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elasticse"
  }
}

logstash kafka 输入不工作

logstash kafka input not working

elasticsearch

apache-kafka

logstash

logstash-configuration