FileChannel 和 ByteBuffer 写入额外数据

Question

我正在创建一个方法，它将接收一个文件并将其拆分为 shardCount 个部分并生成一个奇偶校验文件。

当我运行使用此方法时，似乎我正在将额外的数据写到我的奇偶校验文件中。这是我第一次使用 FileChannel 和 ByteBuffers，所以我不确定我是否完全理解如何使用它们，尽管盯着文档看了大约 8 个小时。

此代码是奇偶校验部分的简化版本。

public static void splitAndGenerateParityFile(File file, int shardCount, String fileID) throws IOException {
    RandomAccessFile rin = new RandomAccessFile(file, "r");
    FileChannel fcin = rin.getChannel();

    //Create parity files
    File parity = new File(fileID + "_parity");
    if (parity.exists()) throw new FileAlreadyExistsException("Could not create parity file! File already exists!");
    RandomAccessFile parityRAF = new RandomAccessFile(parity, "rw");
    FileChannel parityOut = parityRAF.getChannel();

    long bytesPerFile = (long) Math.ceil(rin.length() / shardCount);

    //Make buffers for each section of the file we will be reading from
    for (int i = 0; i < shardCount; i++) {
        ByteBuffer bb = ByteBuffer.allocate(1024);
        shardBuffers.add(bb);
    }

    ByteBuffer parityBuffer = ByteBuffer.allocate(1024);

    //Generate parity
    boolean isParityBufferEmpty = true;
    for (long i = 0; i < bytesPerFile; i++) {
        isParityBufferEmpty = false;
        int pos = (int) (i % 1024);
        byte p = 0;

        if (pos == 0) {
            //Read chunk of file into each buffer
            for (int j = 0; j < shardCount; j++) {
                ByteBuffer bb = shardBuffers.get(j);
                bb.clear();
                fcin.read(bb, bytesPerFile * j + i);
                bb.rewind();
            }
            //Dump parity buffer
            if (i > 0) {
                parityBuffer.rewind();
                parityOut.write(parityBuffer);
                parityBuffer.clear();
                isParityBufferEmpty = true;
            }
        }

        //Get parity
        for (ByteBuffer bb : shardBuffers) {
            if (pos >= bb.limit()) break;
            p ^= bb.get(pos);
        }

        //Put parity in buffer
        parityBuffer.put(pos, p);
    }

    if (!isParityBufferEmpty) {
        parityBuffer.rewind();
        parityOut.write(parityBuffer);
        parityBuffer.clear();
    }

    fcin.close();
    rin.close();
    parityOut.close();
    parityRAF.close();
}

如果奇偶校验算法或文件 IO 有任何问题，或者我可以做些什么来优化它，请告诉我。我很高兴听到其他（更好的）文件 IO 方法。

Answer 1

这是我找到的解决方案（尽管它可能需要更多调整）：

public static void splitAndGenerateParityFile(File file, int shardCount, String fileID) throws IOException {
    int BUFFER_SIZE = 4 * 1024 * 1024;
    RandomAccessFile rin = new RandomAccessFile(file, "r");
    FileChannel fcin = rin.getChannel();

    //Create parity files
    File parity = new File(fileID + "_parity");
    if (parity.exists()) throw new FileAlreadyExistsException("Could not create parity file! File already exists!");
    RandomAccessFile parityRAF = new RandomAccessFile(parity, "rw");
    FileChannel parityOut = parityRAF.getChannel();

    //Create shard files
    ArrayList<File> shards = new ArrayList<>(shardCount);
    for (int i = 0; i < shardCount; i++) {
        File f = new File(fileID + "_part_" + i);
        if (f.exists()) throw new FileAlreadyExistsException("Could not create shard file! File already exists!");
        shards.add(f);
    }

    long bytesPerFile = (long) Math.ceil(rin.length() / shardCount);

    ArrayList<ByteBuffer> shardBuffers = new ArrayList<>(shardCount);

    //Make buffers for each section of the file we will be reading from
    for (int i = 0; i < shardCount; i++) {
        ByteBuffer bb = ByteBuffer.allocate(BUFFER_SIZE);
           shardBuffers.add(bb);
    }

    ByteBuffer parityBuffer = ByteBuffer.allocate(BUFFER_SIZE);

    //Generate parity
    boolean isParityBufferEmpty = true;
    for (long i = 0; i < bytesPerFile; i++) {
        isParityBufferEmpty = false;
        int pos = (int) (i % BUFFER_SIZE);
        byte p = 0;

        if (pos == 0) {
            //Read chunk of file into each buffer
            for (int j = 0; j < shardCount; j++) {
                ByteBuffer bb = shardBuffers.get(j);
                bb.clear();
                fcin.position(bytesPerFile * j + i);
                fcin.read(bb);
                bb.flip();
            }

            //Dump parity buffer
            if (i > 0) {
                parityBuffer.flip();
                while (parityBuffer.hasRemaining()) {
                    parityOut.write(parityBuffer);
                }
                parityBuffer.clear();
                isParityBufferEmpty = true;
            }
        }

        //Get parity
        for (ByteBuffer bb : shardBuffers) {
            if (!bb.hasRemaining()) break;
            p ^= bb.get();
        }

        //Put parity in buffer
        parityBuffer.put(p);
    }

    if (!isParityBufferEmpty) {
        parityBuffer.flip();
        parityOut.write(parityBuffer);
        parityBuffer.clear();
    }

    fcin.close();
    rin.close();
    parityOut.close();
    parityRAF.close();
}

根据 VGR 的建议，我将 rewind() 替换为 flip()。我也切换到相对操作而不是绝对操作。我认为绝对方法不会调整光标位置或限制，所以这可能是错误的原因。我还将缓冲区大小更改为 4MB，因为我有兴趣为大文件生成奇偶校验。

FileChannel 和 ByteBuffer 写入额外数据

FileChannel and ByteBuffer writing extra data

java

io

bytebuffer

filechannel

parity