同一代理中同一主题的 Kafka 多个分区

Question

在深入研究 Kafka 时，我想到了如下问题。

Can Kafka have multiple partitions of the same topic in the same broker?

起初，我认为我会不受欢迎，因为它与分区的目的相矛盾——为了高可用性而进行冗余。但在某种程度上，没有理由不提供这个。

这在 Kafka 中可行吗？如果是这样，这种设置在实践中是否被大量使用？

Answer 1

Can Kafka have multiple partitions of the same topic in the same broker?

是的，完全可以。例如：

╰─$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                          --create --topic rmoff-test \
                          --partitions 6 \
                          --replication-factor 1
Created topic rmoff-test.

╭─rmoff@asgard03 ~/Downloads/kafka_2.13-3.1.0
╰─$ kcat -b localhost:9092 -L
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)
 1 topics:
  topic "rmoff-test" with 6 partitions:
    partition 0, leader 1, replicas: 1, isrs: 1
    partition 1, leader 1, replicas: 1, isrs: 1
    partition 2, leader 1, replicas: 1, isrs: 1
    partition 3, leader 1, replicas: 1, isrs: 1
    partition 4, leader 1, replicas: 1, isrs: 1
    partition 5, leader 1, replicas: 1, isrs: 1

您可能混淆了分区（用于严格排序和高吞吐量）和分区副本的概念（用于 availability/fault 公差）。

您不能在一个代理上有多个副本。如果您尝试创建一个主题，其副本数多于可用代理数，它将失败：

╰─$ # There is one broker running 
╰─$ kcat -b localhost:9092 -L | grep broker
Metadata for all topics (from broker 1: localhost:9092/1):
 1 brokers:
  broker 1 at localhost:9092 (controller)

╰─$ # Now try to create a topic with _one_ partition but _two_ replicas:
╰─$ ./bin/kafka-topics.sh --bootstrap-server localhost:9092 \
                          --create --topic rmoff-test-2 \
                          --partitions 1 \
                          --replication-factor 2                                                          

Error while executing topic command : Unable to replicate the partition 2 time(s): The target replication factor of 2 cannot be reached because only 1 broker(s) are registered.
[2022-03-11 09:51:33,721] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 2 time(s): The target replication factor of 2 cannot be reached because only 1 broker(s) are registered.
 (kafka.admin.TopicCommand$)

同一代理中同一主题的 Kafka 多个分区

Kafka multiple partitions of the same topic in the same broker

apache-kafka