Kafka 生产者说 "unknown_topic_or_partition"
Kafka producer says "unknown_topic_or_partition"
几天来我一直在努力让 kafka-docker 正常工作,但我不知道自己做错了什么。现在,我无法使用 ruby-kafka 客户端访问任何主题,因为节点 "doesn't exist"。这是我的 docker-compose.yml 文件:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9092:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka2:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9093:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 2
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka3:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9094:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 3
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
我指定 "KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'" 因为我想手动创建主题,所以我进入我的第一个代理容器并输入:
./kafka-topics.sh --create --zookeeper 172.19.0.2:2181 --topic test1 --partitions 4 --replication-factor 3
一切似乎都很好:
./kafka-topics.sh --list --zookeeper 172.19.0.2:2181 -> test1
但是,当我尝试这样做时:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test1
它说:
WARN Error while fetching metadata with correlation id 24 : {test1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
如果我再次创建该主题,它说它已经存在,所以我不知道发生了什么。
您需要正确配置网络,因为 Kafka 跨主机工作并且需要能够访问所有主机。
This post详细解释。
您可能还想参考 https://github.com/confluentinc/cp-docker-images/blob/5.0.0-post/examples/cp-all-in-one/docker-compose.yml 的工作示例 Docker Compose。
所以我们在使用 kafka connect 时遇到了这个问题。对此有多种解决方案。修剪所有 docker 图像或更改连接图像中连接配置中的组 ID,如下所示:-
image: debezium/connect:1.1
ports:
- 8083:8083
links:
- schema-registry
environment:
- BOOTSTRAP_SERVERS=kafkaanalytics-mgmt.fptsinternal.com:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
- INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
几天来我一直在努力让 kafka-docker 正常工作,但我不知道自己做错了什么。现在,我无法使用 ruby-kafka 客户端访问任何主题,因为节点 "doesn't exist"。这是我的 docker-compose.yml 文件:
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9092:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka2:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9093:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 2
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9093
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
kafka3:
image: wurstmeister/kafka:0.9.0.1
ports:
- "9094:9092"
links:
- zookeeper
environment:
KAFKA_BROKER_ID: 3
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
KAFKA_ADVERTISED_PORT: 9094
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
volumes:
- /var/run/docker.sock:/var/run/docker.sock
我指定 "KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'" 因为我想手动创建主题,所以我进入我的第一个代理容器并输入:
./kafka-topics.sh --create --zookeeper 172.19.0.2:2181 --topic test1 --partitions 4 --replication-factor 3
一切似乎都很好:
./kafka-topics.sh --list --zookeeper 172.19.0.2:2181 -> test1
但是,当我尝试这样做时:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test1
它说:
WARN Error while fetching metadata with correlation id 24 : {test1=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
如果我再次创建该主题,它说它已经存在,所以我不知道发生了什么。
您需要正确配置网络,因为 Kafka 跨主机工作并且需要能够访问所有主机。
This post详细解释。
您可能还想参考 https://github.com/confluentinc/cp-docker-images/blob/5.0.0-post/examples/cp-all-in-one/docker-compose.yml 的工作示例 Docker Compose。
所以我们在使用 kafka connect 时遇到了这个问题。对此有多种解决方案。修剪所有 docker 图像或更改连接图像中连接配置中的组 ID,如下所示:-
image: debezium/connect:1.1
ports:
- 8083:8083
links:
- schema-registry
environment:
- BOOTSTRAP_SERVERS=kafkaanalytics-mgmt.fptsinternal.com:9092
- GROUP_ID=1
- CONFIG_STORAGE_TOPIC=my_connect_configs
- OFFSET_STORAGE_TOPIC=my_connect_offsets
- STATUS_STORAGE_TOPIC=my_connect_statuses
- INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
- INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter