Kafka 2.1 对保留和 Kafka Stream 应用程序的行为更改，我们可以怎样才能使保留有效？

Question

以下内容来自 2.1 的 Kafka 文档。

https://kafka.apache.org/documentation/

Offset expiration semantics has slightly changed in this version. 
According to the new semantics, offsets of partitions in a group will 
not be removed while the group is subscribed to the corresponding 
topic and is still active (has active consumers). If group becomes 
empty all its offsets will be removed after default offset retention 
period (or the one set by broker) has passed (unless the group becomes 
active again). Offsets associated with standalone (simple) consumers, 
that do not use Kafka group management, will be removed after default 
offset retention period (or the one set by broker) has passed since 
their last commit.

如果我没有理解错的话，只要连接了 Stream Thread 消费者，任何保留设置都不会生效？

我也开始观察stream application重启后出现如下Exception

stream thread - Restoring Stream Tasks failed. Deleting StreamTasks stores to recreate from scratch. org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions:' but stream application uses the property 'StreamsConfig.consumerPrefix(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG), "earliest"'...

我认为它必须与保留有关，但我不知道是什么？

Answer 1

If I understand this correctly, as long as Stream Thread consumer's are connected, no retention setting will be effective?

这仅适用于 __consumer_offset 主题，即 Kafka 内部主题。对于所有 regular/user 个主题，保留时间的应用方式与所有先前版本中的相同。另请注意，这仅适用于将经纪人升级到 2.1.

的情况

关于Streams的日志信息：你不用担心。看来您的应用程序离线时间较长，因此，您的本地商店不再处于一致状态。因此，它已从更改日志主题中删除并从头开始重新创建。

Kafka 2.1 对保留和 Kafka Stream 应用程序的行为更改，我们可以怎样才能使保留有效？

Kafka 2.1 behaviour change for retentions and Kafka Stream application, what can we so that retention works?

apache-kafka-streams