如何从 java 中的大型加密文件中有效地读取给定范围的字节块？
How to effeciently read chunk of bytes of a given range from a large encrypted file in java?
我在服务器中有一个很大的加密文件(10GB+)。我需要将解密后的文件分成小块传输给客户端。当客户端请求一大块字节（比如 18 到 45）时，我必须随机访问文件，读取特定字节，解密并使用 ServletResponseStream 将其传输到客户端。
但由于文件已加密，我必须以 16 字节的块形式读取文件才能正确解密。
因此，如果客户端请求从字节 18 到 45，在服务器中我必须以 16 字节块的倍数读取文件。所以我必须从第 16 字节到第 48 字节随机访问文件。然后解密它。解密后，我必须从第一个字节跳过 2 个字节，从最后一个字节跳过 3 个字节，以 return 客户端请求的适当数据块。
这是我正在尝试做的事情
调整加密文件的开始和结束
long start = 15; // input from client
long end = 45; // input from client
long skipStart = 0; // need to skip for encrypted file
long skipEnd = 0;
// encrypted files, it must be access in blocks of 16 bytes
if(fileisEncrypted){
skipStart = start % 16; // skip 2 byte at start
skipEnd = 16 - end % 16; // skip 3 byte at end
start = start - skipStart; // start becomes 16
end = end + skipEnd; // end becomes 48
}
从头到尾访问加密文件数据
try(final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath())){
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, start, end-start);
// *** No idea how to convert MappedByteBuffer into input stream ***
// InputStream is = (How do I get inputstream for byte 16 to 48 here?)
// the medhod I used earlier to decrypt the all file atonce, now somehow I need the inputstream of specific range
is = new FileEncryptionUtil().getCipherInputStream(is,
EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
// transfering decrypted input stream to servlet response
OutputStream outputStream = response.getOutputStream();
// *** now for chunk transfer, here I also need to
// skip 2 bytes at the start and 3 bytes from the end.
// How to do it? ***/
org.apache.commons.io.IOUtils.copy(is, outputStream)
}
我在上面给出的代码中遗漏了几个步骤。我知道我可以尝试逐字节读取并忽略第一个字节的 2 个字节和最后一个字节的 3 个字节。但我不确定它是否足够有效。此外，客户端可以请求从字节 18 到 2048 的大块，这将需要读取和解密几乎 2 GB 的数据。恐怕创建一个大字节数组会消耗太多内存。
如何在不对服务器处理或内存造成太大压力的情况下高效地完成它？
有什么想法吗？
由于您没有指定您使用的是哪种密码模式，我假设您在 CTR 模式下使用 AES，因为它旨在读取大文件的随机块而不必完全解密它们。
使用 AES-CTR，您可以通过解密代码流式传输文件，并在块可用时将其发送回客户端。所以你只需要内存中 AES 块大小的几个数组，其余的都是从磁盘读取的。您需要添加特殊逻辑来跳过第一个和最后一个块上的一些再见（但您不需要将整个内容加载到内存中）。
在另一个 SO 问题中有一个如何执行此操作的示例（这仅执行 seek）：Seeking in AES-CTR-encrypted input。之后您可以跳过前几个字节，读取到最后一个块并将其调整为您的客户端请求的字节数。
经过一段时间的研究。我就是这样解决的。
首先我创建了一个ByteBufferInputStreamclass。阅读 MappedByteBuffer
public class ByteBufferInputStream extends InputStream {
private ByteBuffer byteBuffer;
public ByteBufferInputStream () {
}
/** Creates a stream with a new non-direct buffer of the specified size. The position and limit of the buffer is zero. */
public ByteBufferInputStream (int bufferSize) {
this(ByteBuffer.allocate(bufferSize));
byteBuffer.flip();
}
/** Creates an uninitialized stream that cannot be used until {@link #setByteBuffer(ByteBuffer)} is called. */
public ByteBufferInputStream (ByteBuffer byteBuffer) {
this.byteBuffer = byteBuffer;
}
public ByteBuffer getByteBuffer () {
return byteBuffer;
}
public void setByteBuffer (ByteBuffer byteBuffer) {
this.byteBuffer = byteBuffer;
}
public int read () throws IOException {
if (!byteBuffer.hasRemaining()) return -1;
return byteBuffer.get();
}
public int read (byte[] bytes, int offset, int length) throws IOException {
int count = Math.min(byteBuffer.remaining(), length);
if (count == 0) return -1;
byteBuffer.get(bytes, offset, count);
return count;
}
public int available () throws IOException {
return byteBuffer.remaining();
}
}
然后通过扩展 InputStream 创建了 BlockInputStream class，这将允许跳过额外的字节并以 16 字节块的倍数读取内部输入流。
public class BlockInputStream extends InputStream {
private final BufferedInputStream inputStream;
private final long totalLength;
private final long skip;
private long read = 0;
private byte[] buff = new byte[16];
private ByteArrayInputStream blockInputStream;
public BlockInputStream(InputStream inputStream, long skip, long length) throws IOException {
this.inputStream = new BufferedInputStream(inputStream);
this.skip = skip;
this.totalLength = length + skip;
if(skip > 0) {
byte[] b = new byte[(int)skip];
read(b);
b = null;
}
}
private int readBlock() throws IOException {
int count = inputStream.read(buff);
blockInputStream = new ByteArrayInputStream(buff);
return count;
}
@Override
public int read () throws IOException {
byte[] b = new byte[1];
read(b);
return (int)b[1];
}
@Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
@Override
public int read (byte[] bytes, int offset, int length) throws IOException {
long remaining = totalLength - read;
if(remaining < 1){
return -1;
}
int bytesToRead = (int)Math.min(length, remaining);
int n = 0;
while(bytesToRead > 0){
if(read % 16 == 0 && bytesToRead % 16 == 0){
int count = inputStream.read(bytes, offset, bytesToRead);
read += count;
offset += count;
bytesToRead -= count;
n += count;
} else {
if(blockInputStream != null && blockInputStream.available() > 0) {
int len = Math.min(bytesToRead, blockInputStream.available());
int count = blockInputStream.read(bytes, offset, len);
read += count;
offset += count;
bytesToRead -= count;
n += count;
} else {
readBlock();
}
}
}
return n;
}
@Override
public int available () throws IOException {
long remaining = totalLength - read;
if(remaining < 1){
return -1;
}
return inputStream.available();
}
@Override
public long skip(long n) throws IOException {
return inputStream.skip(n);
}
@Override
public void close() throws IOException {
inputStream.close();
}
@Override
public synchronized void mark(int readlimit) {
inputStream.mark(readlimit);
}
@Override
public synchronized void reset() throws IOException {
inputStream.reset();
}
@Override
public boolean markSupported() {
return inputStream.markSupported();
}
}
这是我使用这两个 classes
的最终工作实现
private RangeData getRangeData(RangeInfo r) throws IOException, GeneralSecurityException, CryptoException {
// used for encrypted files
long blockStart = r.getStart();
long blockEnd = r.getEnd();
long blockLength = blockEnd - blockStart + 1;
// encrypted files, it must be access in blocks of 16 bytes
if(datafile.isEncrypted()){
blockStart -= blockStart % 16;
blockEnd = blockEnd | 15; // nearest multiple of 16 for length n = ((n−1)|15)+1
blockLength = blockEnd - blockStart + 1;
}
try ( final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath()) )
{
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, blockStart, blockLength);
InputStream inputStream = new ByteBufferInputStream(mappedByteBuffer);
if(datafile.isEncrypted()) {
String encryptionKeyRef = (String) settingsManager.getSetting(AppSetting.DEFAULT_ENCRYPTION_KEY);
inputStream = new FileEncryptionUtil().getCipherInputStream(inputStream,
EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
long skipStart = r.getStart() - blockStart;
inputStream = new BlockInputStream(inputStream, skipStart, r.getLength()); // this will trim the data to n bytes at last
}
return new RangeData(r, inputStream);
}
}
我在服务器中有一个很大的加密文件(10GB+)。我需要将解密后的文件分成小块传输给客户端。当客户端请求一大块字节（比如 18 到 45）时，我必须随机访问文件，读取特定字节，解密并使用 ServletResponseStream 将其传输到客户端。
但由于文件已加密，我必须以 16 字节的块形式读取文件才能正确解密。
因此，如果客户端请求从字节 18 到 45，在服务器中我必须以 16 字节块的倍数读取文件。所以我必须从第 16 字节到第 48 字节随机访问文件。然后解密它。解密后，我必须从第一个字节跳过 2 个字节，从最后一个字节跳过 3 个字节，以 return 客户端请求的适当数据块。
这是我正在尝试做的事情
调整加密文件的开始和结束
long start = 15; // input from client
long end = 45; // input from client
long skipStart = 0; // need to skip for encrypted file
long skipEnd = 0;
// encrypted files, it must be access in blocks of 16 bytes
if(fileisEncrypted){
skipStart = start % 16; // skip 2 byte at start
skipEnd = 16 - end % 16; // skip 3 byte at end
start = start - skipStart; // start becomes 16
end = end + skipEnd; // end becomes 48
}
从头到尾访问加密文件数据
try(final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath())){
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, start, end-start);
// *** No idea how to convert MappedByteBuffer into input stream ***
// InputStream is = (How do I get inputstream for byte 16 to 48 here?)
// the medhod I used earlier to decrypt the all file atonce, now somehow I need the inputstream of specific range
is = new FileEncryptionUtil().getCipherInputStream(is,
EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
// transfering decrypted input stream to servlet response
OutputStream outputStream = response.getOutputStream();
// *** now for chunk transfer, here I also need to
// skip 2 bytes at the start and 3 bytes from the end.
// How to do it? ***/
org.apache.commons.io.IOUtils.copy(is, outputStream)
}
我在上面给出的代码中遗漏了几个步骤。我知道我可以尝试逐字节读取并忽略第一个字节的 2 个字节和最后一个字节的 3 个字节。但我不确定它是否足够有效。此外，客户端可以请求从字节 18 到 2048 的大块，这将需要读取和解密几乎 2 GB 的数据。恐怕创建一个大字节数组会消耗太多内存。
如何在不对服务器处理或内存造成太大压力的情况下高效地完成它？ 有什么想法吗？
由于您没有指定您使用的是哪种密码模式，我假设您在 CTR 模式下使用 AES，因为它旨在读取大文件的随机块而不必完全解密它们。
使用 AES-CTR，您可以通过解密代码流式传输文件，并在块可用时将其发送回客户端。所以你只需要内存中 AES 块大小的几个数组，其余的都是从磁盘读取的。您需要添加特殊逻辑来跳过第一个和最后一个块上的一些再见（但您不需要将整个内容加载到内存中）。
在另一个 SO 问题中有一个如何执行此操作的示例（这仅执行 seek）：Seeking in AES-CTR-encrypted input。之后您可以跳过前几个字节，读取到最后一个块并将其调整为您的客户端请求的字节数。
经过一段时间的研究。我就是这样解决的。
首先我创建了一个ByteBufferInputStreamclass。阅读 MappedByteBuffer
public class ByteBufferInputStream extends InputStream {
private ByteBuffer byteBuffer;
public ByteBufferInputStream () {
}
/** Creates a stream with a new non-direct buffer of the specified size. The position and limit of the buffer is zero. */
public ByteBufferInputStream (int bufferSize) {
this(ByteBuffer.allocate(bufferSize));
byteBuffer.flip();
}
/** Creates an uninitialized stream that cannot be used until {@link #setByteBuffer(ByteBuffer)} is called. */
public ByteBufferInputStream (ByteBuffer byteBuffer) {
this.byteBuffer = byteBuffer;
}
public ByteBuffer getByteBuffer () {
return byteBuffer;
}
public void setByteBuffer (ByteBuffer byteBuffer) {
this.byteBuffer = byteBuffer;
}
public int read () throws IOException {
if (!byteBuffer.hasRemaining()) return -1;
return byteBuffer.get();
}
public int read (byte[] bytes, int offset, int length) throws IOException {
int count = Math.min(byteBuffer.remaining(), length);
if (count == 0) return -1;
byteBuffer.get(bytes, offset, count);
return count;
}
public int available () throws IOException {
return byteBuffer.remaining();
}
}
然后通过扩展 InputStream 创建了 BlockInputStream class，这将允许跳过额外的字节并以 16 字节块的倍数读取内部输入流。
public class BlockInputStream extends InputStream {
private final BufferedInputStream inputStream;
private final long totalLength;
private final long skip;
private long read = 0;
private byte[] buff = new byte[16];
private ByteArrayInputStream blockInputStream;
public BlockInputStream(InputStream inputStream, long skip, long length) throws IOException {
this.inputStream = new BufferedInputStream(inputStream);
this.skip = skip;
this.totalLength = length + skip;
if(skip > 0) {
byte[] b = new byte[(int)skip];
read(b);
b = null;
}
}
private int readBlock() throws IOException {
int count = inputStream.read(buff);
blockInputStream = new ByteArrayInputStream(buff);
return count;
}
@Override
public int read () throws IOException {
byte[] b = new byte[1];
read(b);
return (int)b[1];
}
@Override
public int read(byte[] b) throws IOException {
return read(b, 0, b.length);
}
@Override
public int read (byte[] bytes, int offset, int length) throws IOException {
long remaining = totalLength - read;
if(remaining < 1){
return -1;
}
int bytesToRead = (int)Math.min(length, remaining);
int n = 0;
while(bytesToRead > 0){
if(read % 16 == 0 && bytesToRead % 16 == 0){
int count = inputStream.read(bytes, offset, bytesToRead);
read += count;
offset += count;
bytesToRead -= count;
n += count;
} else {
if(blockInputStream != null && blockInputStream.available() > 0) {
int len = Math.min(bytesToRead, blockInputStream.available());
int count = blockInputStream.read(bytes, offset, len);
read += count;
offset += count;
bytesToRead -= count;
n += count;
} else {
readBlock();
}
}
}
return n;
}
@Override
public int available () throws IOException {
long remaining = totalLength - read;
if(remaining < 1){
return -1;
}
return inputStream.available();
}
@Override
public long skip(long n) throws IOException {
return inputStream.skip(n);
}
@Override
public void close() throws IOException {
inputStream.close();
}
@Override
public synchronized void mark(int readlimit) {
inputStream.mark(readlimit);
}
@Override
public synchronized void reset() throws IOException {
inputStream.reset();
}
@Override
public boolean markSupported() {
return inputStream.markSupported();
}
}
这是我使用这两个 classes
的最终工作实现private RangeData getRangeData(RangeInfo r) throws IOException, GeneralSecurityException, CryptoException {
// used for encrypted files
long blockStart = r.getStart();
long blockEnd = r.getEnd();
long blockLength = blockEnd - blockStart + 1;
// encrypted files, it must be access in blocks of 16 bytes
if(datafile.isEncrypted()){
blockStart -= blockStart % 16;
blockEnd = blockEnd | 15; // nearest multiple of 16 for length n = ((n−1)|15)+1
blockLength = blockEnd - blockStart + 1;
}
try ( final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath()) )
{
MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, blockStart, blockLength);
InputStream inputStream = new ByteBufferInputStream(mappedByteBuffer);
if(datafile.isEncrypted()) {
String encryptionKeyRef = (String) settingsManager.getSetting(AppSetting.DEFAULT_ENCRYPTION_KEY);
inputStream = new FileEncryptionUtil().getCipherInputStream(inputStream,
EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
long skipStart = r.getStart() - blockStart;
inputStream = new BlockInputStream(inputStream, skipStart, r.getLength()); // this will trim the data to n bytes at last
}
return new RangeData(r, inputStream);
}
}