Kafka config "retention.bytes" 是否适用于 compact topic?
Does Kafka config "retention.bytes" apply to compact topic?
“retention.bytes”是否适用于“紧凑”主题?我来这里的原因是在我当前项目的镜头中,我看到分区字节是 2GB,这比配置的“rention.bytes”43MB 大得多。根据文档,似乎没有。
cleanup.policy A string that is either "delete" or "compact" or both.
This string designates the retention policy to use on old log
segments. The default policy ("delete") will discard old segments when
their retention time or size limit has been reached. The "compact"
setting will enable log compaction on the topic.
retention.bytes This configuration controls the maximum size a
partition (which consists of log segments) can grow to before we will
discard old log segments to free up space if we are using the "delete"
retention policy. By default there is no size limit only a time limit.
Since this limit is enforced at the partition level, multiply it by
the number of partitions to compute the topic retention in bytes.
在“cleanup.policy”中,表示当达到“删除”主题的保留大小时,正在发生删除。 “日志压缩”将发生在紧凑的主题上,但没有提及任何条件。
在“retention.bytes”中,只提到了“删除”话题的删除策略,没有提到“压缩”话题。
如果“retention.bytes”不适用于“紧凑”主题,是否意味着它将始终在主题中保留唯一键?如果是这种情况,比如密钥 space 太大,它会消耗大量磁盘 space,我们如何避免这种情况?
最后一个问题:如果以上是真的,我可以说对于一个纯粹的“紧凑”主题,与“删除”和“删除,紧凑”清理策略相比,没有必要设置“retention.bytes " 和 "retention.ms" 呢?
Compaction 会将所有唯一键保留在主题的“头部”,是的,没有限制。尾部未闭合段(参见 segment.bytes
)将保持 un-compacted,直到日志清理器压缩它,然后新的 un-compacted 段的存储使用量将增加。
您可以使用 compact.policy=compact,delete
将 retention.ms
and/or retention.bytes
应用于压缩主题(正如文档所说,“默认情况下没有大小限制,只有时间限制")
“retention.bytes”是否适用于“紧凑”主题?我来这里的原因是在我当前项目的镜头中,我看到分区字节是 2GB,这比配置的“rention.bytes”43MB 大得多。根据文档,似乎没有。
cleanup.policy A string that is either "delete" or "compact" or both. This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. The "compact" setting will enable log compaction on the topic.
retention.bytes This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes.
在“cleanup.policy”中,表示当达到“删除”主题的保留大小时,正在发生删除。 “日志压缩”将发生在紧凑的主题上,但没有提及任何条件。
在“retention.bytes”中,只提到了“删除”话题的删除策略,没有提到“压缩”话题。
如果“retention.bytes”不适用于“紧凑”主题,是否意味着它将始终在主题中保留唯一键?如果是这种情况,比如密钥 space 太大,它会消耗大量磁盘 space,我们如何避免这种情况?
最后一个问题:如果以上是真的,我可以说对于一个纯粹的“紧凑”主题,与“删除”和“删除,紧凑”清理策略相比,没有必要设置“retention.bytes " 和 "retention.ms" 呢?
Compaction 会将所有唯一键保留在主题的“头部”,是的,没有限制。尾部未闭合段(参见 segment.bytes
)将保持 un-compacted,直到日志清理器压缩它,然后新的 un-compacted 段的存储使用量将增加。
您可以使用 compact.policy=compact,delete
将 retention.ms
and/or retention.bytes
应用于压缩主题(正如文档所说,“默认情况下没有大小限制,只有时间限制")