Kafka inside Docker - 如何从命令行 read/write 到一个主题?
Kafka inside Docker - how to read/write to a topic from command line?
我在 Docker 中有一个启用了 SSL 的 Kafka 运行。
docker ps
:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b8f6b1c573a1 nginx:1.13.9-alpine "nginx -g 'daemon of…" 9 hours ago Up 9 hours 80/tcp, 0.0.0.0:8081->443/tcp ng
761ce6ee2960 confluentinc/cp-schema-registry:4.0.0 "/etc/confluent/dock…" 9 hours ago Up 9 hours 0.0.0.0:8080->8080/tcp, 8081/tcp sr
16d7b81dfbc8 confluentinc/cp-kafka:4.0.0 "/etc/confluent/dock…" 9 hours ago Up 9 hours 0.0.0.0:9092-9093->9092-9093/tcp k1
9be579992536 confluentinc/cp-zookeeper:4.0.0 "/etc/confluent/dock…" 9 hours ago Up 9 hours 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zk
如何从命令行写入主题?
已尝试(主题 'test' 存在):
kafka-console-producer --broker-list kafka:9093 --topic test
# [2018-04-23 17:55:14,325] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
# org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. (60s timeout)
kafka-console-producer --broker-list kafka:9092 --topic test
>aa
#[2018-04-23 18:00:59,443] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
#[2018-04-23 18:00:59,444] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
# org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]
kafka-console-producer --broker-list localhost:9092 --topic test dnk306@9801a7a5b33d
>aa
#[2018-04-23 21:52:47,056] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
#[2018-04-23 21:52:47,056] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
#org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]
当在架构注册表或 Kafka 容器中时,如果您使用 Docker 组合配置,这是可能的,因为容器将由 Compose link 编辑,但是,您的容器主机名是可能 k1
而不是 kafka
如果您在 Kafka 容器内或在容器外使用 CLI 命令,那么它只是 localhost:9093
(因为您转发了端口)。
如果你在 Kafka 容器以外的容器中,并且你想通过主机名解析 Kafka 容器,你必须在 docker 运行 处添加 --link kafka
。有关 linking 容器
的信息,请参阅 Docker 文档
同样重要的是,您需要 link 将 Kafka 容器连接到 Zookeeper 容器,并将 Schema Registry 连接到一个或另一个 ZK 或 Kafka,具体取决于它的配置方式。
https://docs.confluent.io/current/installation/docker/docs/quickstart.html
Also SSL Kafka Docker compose examples here
TopicAuthorizationException: Not authorized to access topics: [test]
这表明连接到Kafka 容器成功。您的下一步是 ensure your Java environment has the necessary keys to access the broker over SSL
我在 Docker 中有一个启用了 SSL 的 Kafka 运行。
docker ps
:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b8f6b1c573a1 nginx:1.13.9-alpine "nginx -g 'daemon of…" 9 hours ago Up 9 hours 80/tcp, 0.0.0.0:8081->443/tcp ng
761ce6ee2960 confluentinc/cp-schema-registry:4.0.0 "/etc/confluent/dock…" 9 hours ago Up 9 hours 0.0.0.0:8080->8080/tcp, 8081/tcp sr
16d7b81dfbc8 confluentinc/cp-kafka:4.0.0 "/etc/confluent/dock…" 9 hours ago Up 9 hours 0.0.0.0:9092-9093->9092-9093/tcp k1
9be579992536 confluentinc/cp-zookeeper:4.0.0 "/etc/confluent/dock…" 9 hours ago Up 9 hours 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp zk
如何从命令行写入主题?
已尝试(主题 'test' 存在):
kafka-console-producer --broker-list kafka:9093 --topic test
# [2018-04-23 17:55:14,325] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
# org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. (60s timeout)
kafka-console-producer --broker-list kafka:9092 --topic test
>aa
#[2018-04-23 18:00:59,443] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
#[2018-04-23 18:00:59,444] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
# org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]
kafka-console-producer --broker-list localhost:9092 --topic test dnk306@9801a7a5b33d
>aa
#[2018-04-23 21:52:47,056] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
#[2018-04-23 21:52:47,056] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
#org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [test]
当在架构注册表或 Kafka 容器中时,如果您使用 Docker 组合配置,这是可能的,因为容器将由 Compose link 编辑,但是,您的容器主机名是可能 k1
而不是 kafka
如果您在 Kafka 容器内或在容器外使用 CLI 命令,那么它只是 localhost:9093
(因为您转发了端口)。
如果你在 Kafka 容器以外的容器中,并且你想通过主机名解析 Kafka 容器,你必须在 docker 运行 处添加 --link kafka
。有关 linking 容器
同样重要的是,您需要 link 将 Kafka 容器连接到 Zookeeper 容器,并将 Schema Registry 连接到一个或另一个 ZK 或 Kafka,具体取决于它的配置方式。
https://docs.confluent.io/current/installation/docker/docs/quickstart.html
Also SSL Kafka Docker compose examples here
TopicAuthorizationException: Not authorized to access topics: [test]
这表明连接到Kafka 容器成功。您的下一步是 ensure your Java environment has the necessary keys to access the broker over SSL