在 AWS EC2 上从 HDP 2.4 更新到 HDP 2.5 后无法连接到 kafka

Question

我通过 Virtual box 在 Sandbox 2.4 上实施了一个 kafka producer 项目，但后来在 AWS EC2 上实施了这个项目（使用 Hortonworks cloudbreak）。这导致我升级到 HDP 2.5，因此从 kafka 9 切换到 kafka 10。我进行了必要的依赖和 IP(localhost) 更改，但得到 java.net.ConnectException: Connection timed out: no further information

kafkaBroker内部ip为10.0.75.111，示例kafka生产者如下

public void initialise() {
    Properties props = new Properties();
    props.put("bootstrap.servers", "10.0.75.111:6667");  // replaced localhost
    props.put("acks", "all");
    props.put("retries", 0);
    props.put("batch.size", 16384);
    props.put("linger.ms", 1);
    props.put("buffer.memory", 33554432);
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
    producer = new org.apache.kafka.clients.producer.KafkaProducer<Integer, String>(props);
}

Kafka 侦听器是 PLAINTEXT://localhost:6667

在运行程序返回下面的错误消息后

803862 [kafka-producer-network-thread | producer-14] DEBUG org.apache.kafka.clients.NetworkClient  - Initiating connection to node -1 at 10.0.75.111:6667.
824863 [kafka-producer-network-thread | producer-14] DEBUG org.apache.kafka.common.network.Selector  - Connection with /10.0.75.111 disconnected
java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.kafka.common.network.PlaintextTransportLayer.finishConnect(PlaintextTransportLayer.java:51)
at org.apache.kafka.common.network.KafkaChannel.finishConnect(KafkaChannel.java:73)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:309)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
at java.lang.Thread.run(Thread.java:745)
824863 [kafka-producer-network-thread | producer-14] DEBUG org.apache.kafka.clients.NetworkClient  - Node -1 disconnected.

如有任何帮助，我将不胜感激

Answer 1

如果您要将数据发送到您的 AWS 安装集群，您需要使用外部 IP 地址。但是，如果您运行来自网络，则可以同时使用两者。

在您的情况下，您似乎运行来自网络外部。从内部使用来自 outside/internal 的 public IP 尝试运行ning。那应该让你继续。

在 AWS EC2 上从 HDP 2.4 更新到 HDP 2.5 后无法连接到 kafka

Can't connect to kafka after updating from HDP 2.4 to HDP 2.5 on AWS EC2

amazon-ec2

hortonworks-data-platform

kafka-producer-api