Kafka 侦听器异常：无法完成提交

Question

使用手动 kafka 确认，而 consuming/processing 大量消息，我们观察到以下错误。

如果我们在创建消费者时将max.poll.records 属性设置为1，在处理大量消息时会不会出现性能问题？？？？

2019-10-22 23:01:55.208 UTC [org.springframework.kafka.KafkaListenerEndpointContainer#0-2-C-1] ERROR c.d.s.s.b.a.kafka.KafkaRxTx - Kafka Listner Exception : Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. -> {}
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.sendOffsetCommitRequest(ConsumerCoordinator.java:808)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.commitOffsetsSync(ConsumerCoordinator.java:691)
    at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1416)
    at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:1377)
    at brave.kafka.clients.TracingConsumer.commitSync(TracingConsumer.java:151)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.ackImmediate(KafkaMessageListenerContainer.java:922)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processAck(KafkaMessageListenerContainer.java:904)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.access00(KafkaMessageListenerContainer.java:384)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$ConsumerAcknowledgment.acknowledge(KafkaMessageListenerContainer.java:1593)

Answer 1

是的，它会对性能产生重大影响；多少取决于你的情况；你应该运行测试。

此外，考虑增加 max.poll.interval.ms。

Kafka 侦听器异常：无法完成提交

Kafka Listner Exception : Commit cannot be completed

apache-kafka

kafka-consumer-api

spring-kafka