Appengine 上的 FileBackedOutputStream

Question

我在 Appengine 上的应用程序创建了一个包含更多 65535 行的 csv 文件

但是，我在写入时遇到类型为 OutOfMemoryError 的错误：

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2271)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)

白此码：

public static byte[] joinLines(Collection<String> lines) {
    final ByteArrayOutputStream stream = new ByteArrayOutputStream();

    boolean firstElement = true;

    for (final String part : lines) {
        String value = part + LINE_SEPARATOR;
        if (firstElement) {
            value = addExcelPrefix(value);
            firstElement = false;
        }

        final int currentSize = value.length();
        try {
            stream.write(value.getBytes(ENCODING), 0, currentSize); // OutOfMemoryError HERE
        } catch (UnsupportedEncodingException e) {
            LOGGER.info(e.getMessage());
        }
    }
    return stream.toByteArray();
}

所以我使用Guava的FileBackedOutputStream来解决OutOfMemoryError的问题:

public static byte[] joinLines(Collection<String> lines) throws IOException {
    final FileBackedOutputStream stream = new FileBackedOutputStream(THRESHOLD, true);

    boolean firstElement = true;

    for (final String part : lines) {
        String value = part + LINE_SEPARATOR;
        if (firstElement) {
            value = addExcelPrefix(value);
            firstElement = false;
        }

        final int currentSize = value.length();
        try {
            stream.write(value.getBytes(ENCODING), 0, currentSize);
        } catch (IOException e) {
            LOGGER.error(e.getMessage());
        }
    }

    return stream.asByteSource().read();
}

但是，在 appengine 上，我现在在创建临时文件时出现 SecurityException 类型的错误：

java.lang.SecurityException: Unable to create temporary file
    at java.io.File.checkAndCreate(File.java:2083)
    at java.io.File.createTempFile(File.java:2198)
    at java.io.File.createTempFile(File.java:2244)
    at com.google.common.io.FileBackedOutputStream.update(FileBackedOutputStream.java:196)
    at com.google.common.io.FileBackedOutputStream.write(FileBackedOutputStream.java:178)

如何允许使用 FileBackedOutputStream 在 Appengine 上创建临时文件？
在桶里，怎么样？

谢谢

Answer 1

我使用 GcsService 解决了我的问题：

protected String uploadBytesForCsv(Map<Integer, Map<Integer, Object>> rows) throws IOException {
    LOGGER.info("Get Bytes For Csv");

    final Collection<String> lines = cellsToCsv(rows);
    LOGGER.info("number line : " + lines.size());

    boolean firstElement = true;

    final String fileName = getFileName();

    final GcsFilename gcsFilename = new GcsFilename(config.getBucketName(), fileName);
    final GcsService gcsService = GcsServiceFactory.createGcsService();
    final GcsOutputChannel outputChannel = gcsService.createOrReplace(gcsFilename, GcsFileOptions.getDefaultInstance());

    for (final String part : lines) {
        final ByteArrayOutputStream stream = new ByteArrayOutputStream();
        String value = part + LINE_SEPARATOR;
        if (firstElement) {
            value = addExcelPrefix(value);
            firstElement = false;
        }

        final int currentSize = value.length();
        try {
            stream.write(value.getBytes(ENCODING), 0, currentSize);
            outputChannel.write(ByteBuffer.wrap(stream.toByteArray()));
        } catch (UnsupportedEncodingException e) {
            LOGGER.info(e.getMessage());
        }

        stream.flush();
        stream.close();
    }

    outputChannel.close();

    return new UrlBuilder(config.getStorageUrlForExport())
            .setBucketName(config.getBucketName())
            .setFilename(fileName).build();
}

Appengine 上的 FileBackedOutputStream

FileBackedOutputStream on Appengine

java

google-app-engine

out-of-memory

guava

securityexception