Kubernetes DSE Cassandra CommitLogReplayer$CommitLogReplayException

Question

我已经在 Kubernetes 上安装了 Cassandra (9 pods) 所有 pods 都已启动，运行除了对于一个 pod，显示以下错误。

org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.shouldSkipSegmentOnError(CommitLogReplayer.java:438)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleUnrecoverableError(CommitLogReplayer.java:452)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:109)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:236)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:134)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:154)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:213)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:194)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
    at com.datastax.bdp.DseModule.main(DseModule.java:96)
Caused by: org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:111)
    ... 12 more
ERROR [main] 2021-09-06 06:19:08,990  JVMStabilityInspector.java:251 - JVM state determined to be unstable.  Exiting forcefully due to:
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.shouldSkipSegmentOnError(CommitLogReplayer.java:438)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleUnrecoverableError(CommitLogReplayer.java:452)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:109)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:236)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:134)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:154)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:213)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:194)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
    at com.datastax.bdp.DseModule.main(DseModule.java:96)

有人可以帮我吗

Answer 1

无论出于何种原因，节点上的提交日志段之一已损坏。

您可以通过手动删除 pod 上的此文件来解决此问题：

/var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log

有趣的是，该提交日志段是在 9 月 2 日创建的 (1630582314923)，但您发布的日志条目是从 9 月 6 日开始的。这表明 pod 发生了一些导致文件损坏的事情。

您需要查看 pod 上的 Cassandra 日志（而不是 pod 日志本身）以确定根本原因并加以解决。干杯！

Kubernetes DSE Cassandra CommitLogReplayer$CommitLogReplayException

Kubernetes DSE Cassandra CommitLogReplayer$CommitLogReplayException

cassandra

kubernetes

kubernetes-pod