在 spring kafka 中获取 UNKOWN_PRODUCER_ID 异常

Question

我正在使用 spring boot 2.1.9 和 spring Kafka 2.2.9 以及 Kafka 链式事务。

我每次都收到来自 Kafka producer 的警告。由于这个原因，一些时间功能将不起作用。

我想知道为什么会出现这些错误？配置有问题吗？

2020-05-04 09:12:35.216  WARN [xxxxx-order-service,,,] 10 --- [ad | producer-8] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-8, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-2] Got error produce response with correlation id 1946 on topic-partition process_event-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:35.327  WARN [xxxxx-order-service,,,] 10 --- [ad | producer-8] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-8, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-2] Got error produce response with correlation id 1950 on topic-partition audit-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:53.512  WARN [xxxxx-order-service,,,] 10 --- [ad | producer-6] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-6, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-0] Got error produce response with correlation id 5807 on topic-partition process_submitted_page_count-2, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:53.632  WARN [xxxxx-order-service,,,] 10 --- [ad | producer-6] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-6, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-0] Got error produce response with correlation id 5811 on topic-partition process_event-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID
2020-05-04 09:12:53.752  WARN [xxxxx-order-service,,,] 10 --- [ad | producer-6] o.a.k.clients.producer.internals.Sender  : [Producer clientId=producer-6, transactionalId=xxxxx-Order-Service-JOg4T1vFzW4tuc-0] Got error produce response with correlation id 5816 on topic-partition audit-0, retrying (2147483646 attempts left). Error: UNKNOWN_PRODUCER_ID

Answer 1

我假设您可能会点击这个 issue。

When a streams application has little traffic, then it is possible that consumer purging would delete even the last message sent by a producer (i.e., all the messages sent by this producer have been consumed and committed), and as a result, the broker would delete that producer's ID. The next time when this producer tries to send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case, this error is retriable: the producer would just get a new producer id and retries, and then this time it will succeed.

建议的解决方案：升级 Kafka

现在这个问题已经 solved 版本 2.4.0+ 所以如果你仍然遇到这个问题你需要升级到更新的 Kafka 版本。

替代解决方案：增加保留时间 & transactional.id.expiration.ms

或者，如果您不能（或不想）升级，则可以增加保留期 (log.retention.hours) 以及 transactional.id.expiration.ms，它定义了需要的不活动时间量通过以使生产者被视为过期（默认为 7 天）。

在 spring kafka 中获取 UNKOWN_PRODUCER_ID 异常

Getting UNKOWN_PRODUCER_ID Exception in spring kafka

apache-kafka

kafka-producer-api

spring-kafka