console-producer 连接到节点时出错 kafka1:9092 (org.apache.kafka.clients.NetworkClient) java.net.UnknownHostException
console-producer Error connecting to node kafka1:9092 (org.apache.kafka.clients.NetworkClient) java.net.UnknownHostException
我正在通过 docker-compose 成功启动带有 kafka 的容器。我的意思是成功，我可以登录这样的 kafka 容器并直接从那里生成和使用消息。我还可以看到另一个容器（filebeat）取决于这样的 kafka 容器是否成功连接。
docker-compose.yml
version: '3.2'
services:
zoo1:
image: elevy/zookeeper:latest
environment:
MYID: 1
SERVERS: zoo1
ports:
- "2181:2181"
kafka1:
image: wurstmeister/kafka
command: [start-kafka.sh]
depends_on:
- zoo1
links:
- zoo1
ports:
- "9092:9092"
environment:
KAFKA_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_LOG_RETENTION_HOURS: "168"
KAFKA_LOG_RETENTION_BYTES: "100000000"
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
KAFKA_CREATE_TOPICS: "log:1:1"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
filebeat:
image: docker.elastic.co/beats/filebeat:7.5.2
command: filebeat -e -strict.perms=false
volumes:
- "//c/Users/Cast/megalog-try-1/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "//c/Users/Cast/megalog-try-1/sample-logs:/sample-logs"
links:
- kafka1
depends_on:
- kafka1
然而，当我尝试从我的本地 kafka 向这个容器 kafka 生成一条非常简单的消息时，我收到连接被拒绝的消息。
我发现有人有同样的例外，但情况有点相似。我尝试 没有成功。基本上它指向 "adjust advertised.listeners to be the external IP"。我在我的 docker-compose 中做了它，但没有任何改变。即使这奏效了，我想这也不是最终的解决方案，因为对我来说，因为这个人在 KAFKA_ADVERTISED_LISTENERS
中硬编码了他的计算机 ip 地址
所以我的问题是如何从 Windows 10 中的本地 kafka 运行ning 连接到 kafka 容器以发送消息？
以下是我迄今为止尝试过的所有内容：
1 - 尝试使用本地 ip
ipconfig
...
Endereço IPv4. . . . . . . . . . . . . . . : 192.168.129.97
kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
>[2020-03-02 16:54:44,155] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-03-02 16:54:46,212] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2 - 尝试使用 docker 机器 ip
C:\tools\kafka\bin\windows>docker-machine ip
192.168.99.100
kafka-console-producer.bat --broker-list 192.168.99.100:9092 --topic log
[2020-03-02 17:00:22,500] WARN [Producer clientId=console-producer] Error connecting to node kafka1:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka1
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access0(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
at org.apache.kafka.clients.NetworkClient.access0(NetworkClient.java:69)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1126)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1017)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:538)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
at java.base/java.lang.Thread.run(Thread.java:834)
3 - 尝试使用容器 ip
docker ps
...
3eb114e24b53 wurstmeister/kafka "start-kafka.sh" 0.0.0.0:9092->9092/tcp megalog-try-1_kafka1_1
docker inspect 3eb114e24b53
...
"IPAddress": "172.18.0.4",
C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 172.18.0.4:9092 --topic log
>testing
[2020-03-02 16:57:54,471] WARN [Producer clientId=console-producer] Connection to node -1 (/172.18.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
org.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:888)
*** 根据 CRU 的建议添加
C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_zoo1_1 bash
bash-4.3# ls
bin entrypoint.sh home lib64 mnt root sbin sys usr zookeeper
dev etc lib media proc run srv tmp var
bash-4.3# cd zookeeper/
bash-4.3# ls
bin conf data lib log wal zookeeper-3.4.10.jar
bash-4.3# cd conf
bash-4.3# ls
log4j.properties zoo.cfg
bash-4.3# cat zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zoo1:2888:3888bash-4.3#
来自kafka容器
C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# pwd
/
bash-4.4# ls
bin dev etc home kafka lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
bash-4.4# cd kafka
bash-4.4# ls
kafka-logs-0f385d1d435e
bash-4.4# cd kafka-logs-0f385d1d435e/
bash-4.4# ls
app_logs-0 log-0 meta.properties replication-offset-checkpoint
cleaner-offset-checkpoint log-start-offset-checkpoint recovery-point-offset-checkpoint request_logs-0
bash-4.4# cat meta.properties
#
#Mon Mar 02 21:09:33 GMT 2020
cluster.id=-qitBVmjSUGo7Zd7P5cetw
version=0
broker.id=1
bash-4.4#
*** 添加
在 this article 中偶然发现后，我可以写一个 docker-compose 到 运行 kafka 容器，我的本地 SpringBoot 可以发送消息。尽管如此，我必须在 docker-compose 中手动硬编码 docker-machine ip。好吧，我认为这不是唯一的方法，否则每次我启动 Docker ToolBox 时，我都必须 运行 docker-machine ip 并替换 IP 地址。我想知道是否有更友好的解决方案。
下面的192.168.99.100是我的docker机器ip。
version: "3.2"
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS:
"INTERNAL://kafka:9090,\
EXTERNAL://:9092"
KAFKA_ADVERTISED_LISTENERS:
"INTERNAL://kafka:9090,\
EXTERNAL://192.168.99.100:9092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
"INTERNAL:PLAINTEXT,\
EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
根据Cricket_007的建议，我仔细阅读了another article，但它让我得出结论，我必须使用DNS解析，但我无法在我的[=中设置独占dns解析57=]据我了解的机器。
KAFKA_ADVERTISED_PORT 已弃用。只需使用广告的听众。
您公布的侦听器是 Docker 服务名称，而不是同一 Docker 网络上容器外的任何代码 运行 的可解析地址。
您需要添加 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
只需使用现有的，functional Docker Compose
顺便说一句，links 也是一个已弃用的 Compose 选项
我正在通过 docker-compose 成功启动带有 kafka 的容器。我的意思是成功，我可以登录这样的 kafka 容器并直接从那里生成和使用消息。我还可以看到另一个容器（filebeat）取决于这样的 kafka 容器是否成功连接。
docker-compose.yml
version: '3.2'
services:
zoo1:
image: elevy/zookeeper:latest
environment:
MYID: 1
SERVERS: zoo1
ports:
- "2181:2181"
kafka1:
image: wurstmeister/kafka
command: [start-kafka.sh]
depends_on:
- zoo1
links:
- zoo1
ports:
- "9092:9092"
environment:
KAFKA_LISTENERS: PLAINTEXT://:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_LOG_RETENTION_HOURS: "168"
KAFKA_LOG_RETENTION_BYTES: "100000000"
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181
KAFKA_CREATE_TOPICS: "log:1:1"
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
filebeat:
image: docker.elastic.co/beats/filebeat:7.5.2
command: filebeat -e -strict.perms=false
volumes:
- "//c/Users/Cast/megalog-try-1/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
- "//c/Users/Cast/megalog-try-1/sample-logs:/sample-logs"
links:
- kafka1
depends_on:
- kafka1
然而，当我尝试从我的本地 kafka 向这个容器 kafka 生成一条非常简单的消息时，我收到连接被拒绝的消息。
我发现有人有同样的例外，但情况有点相似。我尝试
所以我的问题是如何从 Windows 10 中的本地 kafka 运行ning 连接到 kafka 容器以发送消息？
以下是我迄今为止尝试过的所有内容：
1 - 尝试使用本地 ip
ipconfig
...
Endereço IPv4. . . . . . . . . . . . . . . : 192.168.129.97
kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 192.168.129.97:9092 --topic log
>[2020-03-02 16:54:44,155] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-03-02 16:54:46,212] WARN [Producer clientId=console-producer] Connection to node -1 (/192.168.129.97:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
2 - 尝试使用 docker 机器 ip
C:\tools\kafka\bin\windows>docker-machine ip
192.168.99.100
kafka-console-producer.bat --broker-list 192.168.99.100:9092 --topic log
[2020-03-02 17:00:22,500] WARN [Producer clientId=console-producer] Error connecting to node kafka1:9092 (id: 1 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: kafka1
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access0(ClusterConnectionStates.java:363)
at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
at org.apache.kafka.clients.NetworkClient.access0(NetworkClient.java:69)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1126)
at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1017)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:538)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:335)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
at java.base/java.lang.Thread.run(Thread.java:834)
3 - 尝试使用容器 ip
docker ps
...
3eb114e24b53 wurstmeister/kafka "start-kafka.sh" 0.0.0.0:9092->9092/tcp megalog-try-1_kafka1_1
docker inspect 3eb114e24b53
...
"IPAddress": "172.18.0.4",
C:\tools\kafka\bin\windows>kafka-console-producer.bat --broker-list 172.18.0.4:9092 --topic log
>testing
[2020-03-02 16:57:54,471] WARN [Producer clientId=console-producer] Connection to node -1 (/172.18.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
org.apache.kafka.common.KafkaException: Producer closed while send in progress
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:888)
*** 根据 CRU 的建议添加
C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_zoo1_1 bash
bash-4.3# ls
bin entrypoint.sh home lib64 mnt root sbin sys usr zookeeper
dev etc lib media proc run srv tmp var
bash-4.3# cd zookeeper/
bash-4.3# ls
bin conf data lib log wal zookeeper-3.4.10.jar
bash-4.3# cd conf
bash-4.3# ls
log4j.properties zoo.cfg
bash-4.3# cat zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/zookeeper/data
dataLogDir=/zookeeper/wal
#snapCount=100000
autopurge.purgeInterval=1
clientPort=2181
quorumListenOnAllIPs=true
server.1=zoo1:2888:3888bash-4.3#
来自kafka容器
C:\tools\kafka\bin\windows>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# pwd
/
bash-4.4# ls
bin dev etc home kafka lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
bash-4.4# cd kafka
bash-4.4# ls
kafka-logs-0f385d1d435e
bash-4.4# cd kafka-logs-0f385d1d435e/
bash-4.4# ls
app_logs-0 log-0 meta.properties replication-offset-checkpoint
cleaner-offset-checkpoint log-start-offset-checkpoint recovery-point-offset-checkpoint request_logs-0
bash-4.4# cat meta.properties
#
#Mon Mar 02 21:09:33 GMT 2020
cluster.id=-qitBVmjSUGo7Zd7P5cetw
version=0
broker.id=1
bash-4.4#
*** 添加
在 this article 中偶然发现后，我可以写一个 docker-compose 到 运行 kafka 容器，我的本地 SpringBoot 可以发送消息。尽管如此，我必须在 docker-compose 中手动硬编码 docker-machine ip。好吧，我认为这不是唯一的方法，否则每次我启动 Docker ToolBox 时，我都必须 运行 docker-machine ip 并替换 IP 地址。我想知道是否有更友好的解决方案。
下面的192.168.99.100是我的docker机器ip。
version: "3.2"
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS:
"INTERNAL://kafka:9090,\
EXTERNAL://:9092"
KAFKA_ADVERTISED_LISTENERS:
"INTERNAL://kafka:9090,\
EXTERNAL://192.168.99.100:9092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:
"INTERNAL:PLAINTEXT,\
EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
根据Cricket_007的建议，我仔细阅读了another article，但它让我得出结论，我必须使用DNS解析，但我无法在我的[=中设置独占dns解析57=]据我了解的机器。
KAFKA_ADVERTISED_PORT已弃用。只需使用广告的听众。您公布的侦听器是 Docker 服务名称，而不是同一 Docker 网络上容器外的任何代码 运行 的可解析地址。
您需要添加
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP只需使用现有的，functional Docker Compose
顺便说一句，links 也是一个已弃用的 Compose 选项