Apache Camel：如何从 AWS 流式传输大文件？

Question

我有如下路线。它适用于小文件，但对于大文件（大约 6GB 或更多），我的程序会耗尽内存。如何在不存储在内存中的情况下流式传输内容？

public void configure() throws Exception {
    S3LastModifiedFilter lastModifiedFilter =
            new S3LastModifiedFilter(s3Properties.getLastModifiedWithinSeconds(), s3Properties.getPrefix());

    from(inboundS3Uri)
            .filter().method(lastModifiedFilter, "accept")
            .idempotentConsumer(header(S3Constants.KEY),
                    MemoryIdempotentRepository.memoryIdempotentRepository(inboundCacheSize))
            .convertBodyTo(byte[].class, UTF_8.name())
            .process(new ForHttpMessageProcessor(httpProperties))
            .to(outboundHttpUri);
}

错误:

Caused by: org.apache.camel.TypeConversionException: Error during type conversion from type: java.lang.String to the required type: byte[] with value [Body is instance of java.io.InputStream] due java.lang.OutOfMemoryError: Java heap space
    at org.apache.camel.impl.converter.BaseTypeConverterRegistry.createTypeConversionException(BaseTypeConverterRegistry.java:629)
    at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:190)
    at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:108)
    ... 23 common frames omitted
Caused by: org.apache.camel.RuntimeCamelException: java.lang.OutOfMemoryError: Java heap space
    at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1774)

Answer 1

原来是 convertBodyTo(byte[].class, UTF_8.name()) 行出了问题。它试图缓冲内存中的内容并转换为字符串。我把它注释掉了，现在代码可以运行了。

Apache Camel：如何从 AWS 流式传输大文件？

Apache Camel: How to stream large files from AWS?

apache-camel

apache-camel-aws