如何在不重启 cassandra 节点的情况下重建 sstable?
How can I rebuild a sstable without restart the cassandra node?
当我在一个节点上做一个紧凑的工作时,它会抛出以下异常:
ERROR [CompactionExecutor:116922] 2016-04-07 12:51:17,291 CassandraDaemon.java:153 - Exception in thread Thread[CompactionExecutor:116922,1,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/data1/data/cassandra_uc_log/log_user-2fdda2a03a7f11e58156c78e55b68188/cassandra_uc_log-log_user-ka-7611-Data.db): corruption detected, chunk at 602529 of l
ength 12126.
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:92) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:41) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:326) ~[apache-cassandra-2.1.2.jar:2.1.2]
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444) ~[na:1.7.0_60]
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) ~[na:1.7.0_60]
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:311) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:132) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.AbstractCell.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.AbstractCell.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at com.google.common.collect.Iterators.computeNext(Iterators.java:645) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na]
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na]
at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:110) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:200) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:115) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:183) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:2.1.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/data1/data/cassandra_uc_log/log_user-2fdda2a03a7f11e58156c78e55b68188/cassandra_uc_log-log_user-ka-7611-Data.db): corruption detected, chunk at 602529 of length 12126.
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:112) ~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) ~[apache-cassandra-2.1.2.jar:2.1.2]
... 37 common frames omitted
所以,我删除了 sstable cassandra_uc_log-log_user-ka-7611-Data.db 而不是修复,但它抛出了一个新的异常:
Repair session f33a4b10-ffb7-11e5-8fe3-31b2e5b5b0b2 for range (-5651751204441903619,-5621122634670931727] failed with error java.io.IOException: Failed during snapshot creation.
如何修复或重建 sstable cassandra_uc_log-log_user-ka-7611-Data.db
恐怕您必须重新启动已删除文件的节点。下次您还可以尝试 运行 nodetool scrub 损坏的 sstable 就位。
您没有提到您的 Cassandra 版本,但假设您对 Cassandra 2.1+ 使用增量修复,您可能还必须进行全面修复,以防 sstable 已经处于修复状态。
当我在一个节点上做一个紧凑的工作时,它会抛出以下异常:
ERROR [CompactionExecutor:116922] 2016-04-07 12:51:17,291 CassandraDaemon.java:153 - Exception in thread Thread[CompactionExecutor:116922,1,main] org.apache.cassandra.io.sstable.CorruptSSTableException: org.apache.cassandra.io.compress.CorruptBlockException: (/data1/data/cassandra_uc_log/log_user-2fdda2a03a7f11e58156c78e55b68188/cassandra_uc_log-log_user-ka-7611-Data.db): corruption detected, chunk at 602529 of l ength 12126. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:92) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:41) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:326) ~[apache-cassandra-2.1.2.jar:2.1.2] at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444) ~[na:1.7.0_60] at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) ~[na:1.7.0_60] at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:311) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:132) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:86) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.AbstractCell.computeNext(AbstractCell.java:52) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.AbstractCell.computeNext(AbstractCell.java:46) ~[apache-cassandra-2.1.2.jar:2.1.2] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:116) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at com.google.common.collect.Iterators.computeNext(Iterators.java:645) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) ~[guava-16.0.jar:na] at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:110) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:200) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:115) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:183) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:232) ~[apache-cassandra-2.1.2.jar:2.1.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60] at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60] Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/data1/data/cassandra_uc_log/log_user-2fdda2a03a7f11e58156c78e55b68188/cassandra_uc_log-log_user-ka-7611-Data.db): corruption detected, chunk at 602529 of length 12126. at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:112) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) ~[apache-cassandra-2.1.2.jar:2.1.2] ... 37 common frames omitted
所以,我删除了 sstable cassandra_uc_log-log_user-ka-7611-Data.db 而不是修复,但它抛出了一个新的异常:
Repair session f33a4b10-ffb7-11e5-8fe3-31b2e5b5b0b2 for range (-5651751204441903619,-5621122634670931727] failed with error java.io.IOException: Failed during snapshot creation.
如何修复或重建 sstable cassandra_uc_log-log_user-ka-7611-Data.db
恐怕您必须重新启动已删除文件的节点。下次您还可以尝试 运行 nodetool scrub 损坏的 sstable 就位。
您没有提到您的 Cassandra 版本,但假设您对 Cassandra 2.1+ 使用增量修复,您可能还必须进行全面修复,以防 sstable 已经处于修复状态。