filebeat 到 kafka：无法连接到代理
filebeat to kafka : Failed to connect to broker
我是 apache 环境中的新手，目前正在尝试将日志数据从 filebeat 生产者发送到 kafka 代理。
环境：
- kafka 2.11（通过ambari安装）
- filebeat 7.4.2 (windows)
我试图将日志从 filebeat 发送到 ambari，我已经启动了 kafka 服务器并创建了名为 "test" 的主题，它列在 --list 上。我对卡夫卡经纪人的港口感到很困惑。在一些教程中，我看到他们使用的是 9092 而不是 2181。那么现在，我应该使用哪个端口从 filebeat 发送日志？
这是我的 filebeat.conf
filebeat.inputs:
- type: log
paths:
- C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
hosts: ["x.x.x.x:9092"]
topic: "test"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
结果
2020-06-10T09:00:32.214+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:32.214+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)
2020-06-10T09:00:32.466+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:34.475+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:34.475+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:34.477+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:34.477+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:34.478+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)
2020-06-10T09:00:34.729+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:36.737+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:36.737+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)
2020-06-10T09:00:36.989+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:39.002+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:39.002+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
这让我想知道我是否真的有 9092 端口。所以我查看了我最关心的server.properties.Some：
port=6667
listeners=PLAINTEXT://x.x.x.x:6667
所以我再次尝试执行 filebeat.conf 并将端口 9092 更改为 6667，结果如下
2020-06-10T09:18:01.448+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667
2020-06-10T09:18:01.450+0700 INFO kafka/log.go:53 producer/broker/1001 starting up
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 producer/broker/1001 state change to [open] on test/0
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 producer/leader/test/0 selected broker 1001
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.452+0700 INFO kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.453+0700 DEBUG [kafka] kafka/client.go:264 finished kafka batch
2020-06-10T09:18:01.453+0700 DEBUG [kafka] kafka/client.go:278 Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [normal]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 abandoning broker 1001
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/broker/1001 shut down
问题
发生了什么事？应该使用哪个端口？每个端口的用途是什么？
如有任何回复，我们将不胜感激。谢谢
更新
根据此 source 正确的来源是 6667，因为 kafka 是通过 ambari 安装的
对可以使用的端口没有限制，仅取决于可用性。
在第一种情况下，正如您所说，代理可能已经在 6667 上启动，因此在 9092 上没有进程 运行。
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 Failed to
connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host
接下来，当你提到advertised.listeners属性时，你应该确保你在advertised.listeners中提到的IP是分配给那台机器的IP。您不能分配 1.1.1.1:9092（仅举一些例子）。
执行ifconfig (linux), ipconfig (windows) 并在可从您的应用程序计算机访问的网络接口上查看您计算机的IP。
在 linux 中，它将 大部分 是 eth0
此 IP 必须可以从您所在的计算机访问 运行 您的应用程序。
因此，您的应用程序所在的机器 运行 应该能够解析该 IP。您可能还想检查您的 Kafka 代理和您 运行 您的应用程序所在的机器之间的网络连接。
我是 apache 环境中的新手，目前正在尝试将日志数据从 filebeat 生产者发送到 kafka 代理。
环境：
- kafka 2.11（通过ambari安装）
- filebeat 7.4.2 (windows)
我试图将日志从 filebeat 发送到 ambari，我已经启动了 kafka 服务器并创建了名为 "test" 的主题，它列在 --list 上。我对卡夫卡经纪人的港口感到很困惑。在一些教程中，我看到他们使用的是 9092 而不是 2181。那么现在，我应该使用哪个端口从 filebeat 发送日志？
这是我的 filebeat.conf
filebeat.inputs:
- type: log
paths:
- C:/Users/A/Desktop/DATA/mailbox3.csv
output.kafka:
hosts: ["x.x.x.x:9092"]
topic: "test"
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
结果
2020-06-10T09:00:32.214+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:32.214+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:32.215+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (3 attempts remaining)
2020-06-10T09:00:32.466+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:34.475+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:34.475+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:34.477+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:34.477+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:34.478+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining)
2020-06-10T09:00:34.729+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:36.737+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:36.737+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:36.738+0700 INFO kafka/log.go:53 client/metadata retrying after 250ms... (1 attempts remaining)
2020-06-10T09:00:36.989+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
2020-06-10T09:00:39.002+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:9092: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:39.002+0700 INFO kafka/log.go:53 client/metadata got error from broker -1 while fetching metadata: dial tcp x.x.x.x:9092: connectex: No connection could be made because the target machine actively refused it.
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 kafka message: client/metadata no available broker to send metadata request to
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 client/brokers resurrecting 1 dead seed brokers
2020-06-10T09:00:39.004+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:9092
这让我想知道我是否真的有 9092 端口。所以我查看了我最关心的server.properties.Some：
port=6667
listeners=PLAINTEXT://x.x.x.x:6667
所以我再次尝试执行 filebeat.conf 并将端口 9092 更改为 6667，结果如下
2020-06-10T09:18:01.448+0700 INFO kafka/log.go:53 client/metadata fetching metadata for [test] from broker x.x.x.x:6667
2020-06-10T09:18:01.450+0700 INFO kafka/log.go:53 producer/broker/1001 starting up
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 producer/broker/1001 state change to [open] on test/0
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 producer/leader/test/0 selected broker 1001
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.452+0700 INFO kafka/log.go:53 producer/broker/1001 state change to [closing] because dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.453+0700 DEBUG [kafka] kafka/client.go:264 finished kafka batch
2020-06-10T09:18:01.453+0700 DEBUG [kafka] kafka/client.go:278 Kafka publish failed with: dial tcp: lookup x.x.x.x: no such host
2020-06-10T09:18:01.454+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [flushing-3]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [normal]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 state change to [retrying-3]
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/leader/test/0 abandoning broker 1001
2020-06-10T09:18:01.456+0700 INFO kafka/log.go:53 producer/broker/1001 shut down
问题
发生了什么事？应该使用哪个端口？每个端口的用途是什么？
如有任何回复，我们将不胜感激。谢谢
更新
根据此 source 正确的来源是 6667，因为 kafka 是通过 ambari 安装的
对可以使用的端口没有限制，仅取决于可用性。 在第一种情况下，正如您所说，代理可能已经在 6667 上启动，因此在 9092 上没有进程 运行。
2020-06-10T09:18:01.451+0700 INFO kafka/log.go:53 Failed to connect to broker x.x.x.x:6667: dial tcp: lookup x.x.x.x: no such host
接下来，当你提到advertised.listeners属性时，你应该确保你在advertised.listeners中提到的IP是分配给那台机器的IP。您不能分配 1.1.1.1:9092（仅举一些例子）。
执行ifconfig (linux), ipconfig (windows) 并在可从您的应用程序计算机访问的网络接口上查看您计算机的IP。
在 linux 中，它将 大部分 是 eth0
此 IP 必须可以从您所在的计算机访问 运行 您的应用程序。 因此，您的应用程序所在的机器 运行 应该能够解析该 IP。您可能还想检查您的 Kafka 代理和您 运行 您的应用程序所在的机器之间的网络连接。