Spring 分区的 kafkaListener 消费者组 ID

Question

目前，我们为给定的主题创建多个kafka消费者，主题中的所有消息都由同一消费者组中的所有消费者一起处理。

我们现在想加强这一点。我们希望同一消费者组中的多个消费者从主题的指定分区读取，而不是创建多个 kaka consumers/listeners 从同一主题读取。

可能吗？我们是否有任何参考可以在我们的每个 kafka 侦听器中一起指定消费者组 ID 和分区？

Answer 1

如果我理解你的要求，你必须使用手动分区分配。参见 https://docs.spring.io/spring-kafka/docs/current/reference/html/#manual-assignment

You can also configure POJO listeners with explicit topics and partitions (and, optionally, their initial offsets). The following example shows how to do so:

@KafkaListener(groupId = "thing2", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
          @TopicPartition(topic = "topic2", partitions = { "0", "1" })
        })

@KafkaListener(groupId = "thing2", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "2", "3" }),
          @TopicPartition(topic = "topic2", partitions = { "2", "3" })
        })

等等

Spring 分区的 kafkaListener 消费者组 ID

Spring kafkaListener Consumer GroupID for a Partition

apache-kafka

spring-kafka