kafka-connect 分布式工作者可以使用主题 'connect-offset' 和配置 'cleanup.policy=compact,delete'

could kafka-connect distributed worker work with topic 'connect-offset' with configuration 'cleanup.policy=compact,delete'

我启动了一个 kafka-connect 分布式工作集群,它使用主题 connect-offset 进行偏移存储:

offset.storage.topic=connect-offset

由于为代理提供了默认策略 'cleanup.policy=delete',因此在使用 'cleanup.policy=compact' 创建主题时,我最终将获得此主题的 'cleanup.policy=compact,delete'。这导致 kafka-connect 工作进程抛出异常:

org.apache.kafka.common.config.ConfigException: Topic 'slpe-connect-offset' supplied via the 'offset.storage.topic' property is required to have 'cleanup.policy=compact' to guarantee consistency and durability of source connector offsets, but found the topic currently has 'cleanup.policy=compact,delete'. Continuing would likely result in eventually losing source connector offsets and problems restarting this Connect cluster in the future. Change the 'offset.storage.topic' property in the Connect worker configurations to use a topic with 'cleanup.policy=compact'.

问题:是否有任何kafka-connect worker配置允许吃掉这个异常来保持worker进程?虽然这是一个风险,但 delete 在达到保留或大小限制之前不会发生。

此验证在 KAFKA-9216 (PR) 中引入并影响 2.3.2、2.6.0、2.4.2、2.5.1 版本:

The current change helps guard against users running Connect with topics that have delete cleanup policy enabled, which will remove all connector configurations, source offsets, and connector & task statuses that are older than the retention time. This means that, for example, the configuation for a long-running connector could be deleted by the broker, and this will cause restart issues upon a subsequent rebalance or restarting of Connect worker(s). Connect behavior requires that its internal topics are compacted and not deleted after some retention time

因此,非常不推荐对 Kafka Connect 内部主题使用删除干净策略:只需重新创建这些主题。

没有任何配置 属性 可以忽略此验证,但如果需要,您可以将 Kafka Connect 降级到没有它的版本。