kafka-reassign-partitions --generate for topic __commit_offsets 给我一个奇怪的结果：分区副本只在一个代理上

Question

kafka-reassign-partitions --generate for topic __commit_offsets 给了我奇怪的结果：分区副本只在一个代理上，但每个分区预计有 3 个副本。

我执行以下操作：

test@kafka-1:~/Kafka-Docker$ sudo docker exec -it broker bash
[appuser@broker ~]$ echo '{"version":1, "topics":[{"topic":"__consumer_offsets"}]}' > topics-to-move.json
[appuser@broker ~]$ kafka-reassign-partitions --zookeeper $KAFKA_ZOOKEEPER_CONNECT --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3" --generate
Warning: --zookeeper is deprecated, and will be removed in a future version of Kafka.
Current partition replica assignment
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[1],"log_dirs":["any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"__consumer_offsets","partition":0,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":1,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":2,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":3,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":4,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":5,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":6,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":7,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":8,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":9,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":10,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":11,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":12,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":13,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":14,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":15,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":16,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":17,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":18,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":19,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":20,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":21,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":22,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":23,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":24,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":25,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":26,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":27,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":28,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":29,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":30,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":31,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":32,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":33,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":34,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":35,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":36,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":37,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":38,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":39,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":40,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":41,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":42,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":43,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":44,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":45,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":46,"replicas":[2],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":47,"replicas":[3],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":48,"replicas":[1],"log_dirs":["any"]},{"topic":"__consumer_offsets","partition":49,"replicas":[2],"log_dirs":["any"]}]}
[appuser@broker ~]$

我看到 zookeepers 选项已被弃用，所以我也尝试了这个（结果相同）：

kafka-reassign-partitions --bootstrap-server kafka-1:9092  --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3" --generate

问题：

如您所见，“当前分区副本分配”只有“副本：[1]”，“提议的分区重新分配配置”有“副本：[1]”或“副本：[2]”或“副本：每个分区 [3]"。

但是对于“提议的分区重新分配配置”，它应该是类似于“副本：[1,2,3]”或“副本：[2,3,1]”等等的冗余 __commit_offsets 跨越 3 个经纪人。

为什么每个分区只有一个副本只绑定到一个代理？为什么每个分区的 3 个副本没有分布在 3 个代理中？

在所有 3 个代理中复制的好主题示例（但它只有一个分区），它看起来是这样的：

{"version":1,"partitions":[{"topic":"test_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]}]}

我的设置： 3 个 VM 192.168.1.11、.12 和 .13 每个都有 zookeeper 和 broker，zookeepers 连接到 quorum ok - 我通过它的 REST API 看到只有一个领导者（例如 http://192.168.1.11:4888/commands/stats 与 .12 和 .12 相同。 13 个 IP），这很好。还测试了关闭其中一名动物园管理员和在这种情况下选出的新领导者 - 完美无缺。

192.168.1.11 VM 的 docker-compose.yml 示例是（其他类似）：

version: '3.7'

x-zoo: &zoo "kafka-1:2888:3888;kafka-2:2888:3888;kafka-3:2888:3888"
x-kafkaZookeepers: &kafkaZookeepers "kafka-1:2181,kafka-2:2181,kafka-3:2181"
x-kafkaBrokers: &kafkaBrokers "kafka-1:9092,kafka-2:9092,kafka-3:9092"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
      - "4888:8080"
    extra_hosts:
      - "kafka-1:192.168.1.11"
      - "kafka-2:192.168.1.12"
      - "kafka-3:192.168.1.13"
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_QUORUM_LISTEN_ON_ALL_IPS: 'true'
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_PEER_PORT: 2888
      ZOOKEEPER_LEADER_PORT: 3888
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: *zoo
    volumes:
      - ./kafka-data/zookeeper:/var/lib/zookeeper/data
      - ./kafka-data/zookeeper-logs:/var/lib/zookeeper/log
    networks:
      - mynet

  broker:
    image: confluentinc/cp-kafka:6.2.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    extra_hosts:
      - "kafka-1:192.168.1.11"
      - "kafka-2:192.168.1.12"
      - "kafka-3:192.168.1.13"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_LISTENERS: OUTSIDE://0.0.0.0:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: OUTSIDE://192.168.1.11:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: OUTSIDE
      KAFKA_ZOOKEEPER_CONNECT: *kafkaZookeepers
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_DEFAULT_REPLICATION_FACTOR: 3
      KAFKA_LOG_MESSAGE_TIMESTAMP_TYPE: 'LogAppendTime'
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONNECTIONS_MAX_IDLE_MS: 31536000000 # 1 year
    volumes:
      - ./kafka-data/kafka:/var/lib/kafka/data
    networks:
      - mynet

networks:
  mynet:
    driver: bridge

首先设置为：

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_DEFAULT_REPLICATION_FACTOR: 1

然后在 .12 和 .13 虚拟机上添加了 +2 个其他设置（当然有不同的 ZOOKEEPER_SERVER_ID 和 KAFKA_BROKER_ID) 并且每个 KAFKA_DEFAULT_REPLICATION_FACTOR 和 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR 从 1 更改为 3.

然后新主题创建成功，复制因子为 3，我也可以轻松地将现有非系统主题的复制因子更改为 3，但当然主题 __commit_offsets 除外 - 它需要使用重新分配kafka-reassign-partitions 命令行工具。

Answer 1

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR changed from 1 to 3.

偏移量主题仅在全新集群上创建。如果您修改 offsets.topic.replication.factor.

，它不会追溯更新

new topics created OK with replication factor 3

正确，但是，在您增加default.replication.factor之前创建的任何主题也将只有一个副本。

重新分配分区工具没有考虑您的服务器属性。您要么需要不同的工具，要么自己手动放置副本。 -

kafka-reassign-partitions --generate for topic __commit_offsets 给我一个奇怪的结果：分区副本只在一个代理上

kafka-reassign-partitions --generate for topic __commit_offsets gives me strange result: partition replica only on one broker anyway

replication

apache-kafka