添加新分区时的 Kafka 消息排序保证
Kafka Message Ordering Guarantees When New Partition Added
我正在评估不同的 streaming/messaging 服务以用作事件总线。我正在考虑的维度之一是每项服务提供的订购保证。我正在探索的两个选项是 AWS Kinesis 和 Kafka，从高层次上看，它们似乎都提供类似的排序保证，保证记录可以按照它们发布的相同顺序使用 仅在shard/partition.
似乎 AWS Kinesis API 暴露 the ids of the parent shard(s) such that Consumer Groups using KCL 可以确保具有相同分区键的记录可以按照它们发布的顺序使用（假设是单线程发布者），即使分片正在拆分和合并。
我的问题是，Kafka是否提供任何类似的功能，即使在发布消息时添加了分区，也可以按顺序使用使用特定键发布的记录？根据我的阅读，我对分区选择的理解（如果您在记录中指定键）的行为类似于 HASH(key) % PARTITION_COUNT。因此，如果添加了额外的分区，它们将发布具有特定密钥的所有消息的分区可能会（并且我已经证明它在本地）发生变化。同时，组 Coordinator/Leader 将在接收来自该主题的记录的消费者组中的消费者之间重新分配分区所有权。但是，重新分配后，将在两个不同的分区中找到具有相同键的记录（可能未使用的记录）。那么，从Consumer Group层面是不是就没有办法保证现在在不同分区中找到的相同key的未消费记录会按照发布的顺序被消费呢？
我对这两种服务的经验都很少，所以我的理解可能有缺陷。任何建议表示赞赏！
我的理解是正确的（由@OneCricketeer 和 the documentation 证实）。这是文档的相关部分：
Although it’s possible to increase the number of partitions over time, one has to be careful if messages are produced with keys. When publishing a keyed message, Kafka deterministically maps the message to a partition based on the hash of the key. This provides a guarantee that messages with the same key are always routed to the same partition. This guarantee can be important for certain applications since messages within a partition are always delivered in order to the consumer. If the number of partitions changes, such a guarantee may no longer hold. To avoid this situation, a common practice is to over-partition a bit. Basically, you determine the number of partitions based on a future target throughput, say for one or two years later. Initially, you can just have a small Kafka cluster based on your current throughput. Over time, you can add more brokers to the cluster and proportionally move a subset of the existing partitions to the new brokers (which can be done online). This way, you can keep up with the throughput growth without breaking the semantics in the application when keys are used.
我正在评估不同的 streaming/messaging 服务以用作事件总线。我正在考虑的维度之一是每项服务提供的订购保证。我正在探索的两个选项是 AWS Kinesis 和 Kafka，从高层次上看，它们似乎都提供类似的排序保证，保证记录可以按照它们发布的相同顺序使用 仅在shard/partition.
似乎 AWS Kinesis API 暴露 the ids of the parent shard(s) such that Consumer Groups using KCL 可以确保具有相同分区键的记录可以按照它们发布的顺序使用（假设是单线程发布者），即使分片正在拆分和合并。
我的问题是，Kafka是否提供任何类似的功能，即使在发布消息时添加了分区，也可以按顺序使用使用特定键发布的记录？根据我的阅读，我对分区选择的理解（如果您在记录中指定键）的行为类似于 HASH(key) % PARTITION_COUNT。因此，如果添加了额外的分区，它们将发布具有特定密钥的所有消息的分区可能会（并且我已经证明它在本地）发生变化。同时，组 Coordinator/Leader 将在接收来自该主题的记录的消费者组中的消费者之间重新分配分区所有权。但是，重新分配后，将在两个不同的分区中找到具有相同键的记录（可能未使用的记录）。那么，从Consumer Group层面是不是就没有办法保证现在在不同分区中找到的相同key的未消费记录会按照发布的顺序被消费呢？
我对这两种服务的经验都很少，所以我的理解可能有缺陷。任何建议表示赞赏！
我的理解是正确的（由@OneCricketeer 和 the documentation 证实）。这是文档的相关部分：
Although it’s possible to increase the number of partitions over time, one has to be careful if messages are produced with keys. When publishing a keyed message, Kafka deterministically maps the message to a partition based on the hash of the key. This provides a guarantee that messages with the same key are always routed to the same partition. This guarantee can be important for certain applications since messages within a partition are always delivered in order to the consumer. If the number of partitions changes, such a guarantee may no longer hold. To avoid this situation, a common practice is to over-partition a bit. Basically, you determine the number of partitions based on a future target throughput, say for one or two years later. Initially, you can just have a small Kafka cluster based on your current throughput. Over time, you can add more brokers to the cluster and proportionally move a subset of the existing partitions to the new brokers (which can be done online). This way, you can keep up with the throughput growth without breaking the semantics in the application when keys are used.