从mybatis读取数据时OOM

Question

我在尝试使用 MyBatis 读取大型结果集时收到 OOM 错误。 MyBatis 版本 3.5 我想遍历游标而不是加载整个查询结果。有没有人有一个从 mybatis select 而不是整个结果返回游标的工作示例。

[java] SEVERE:Memory usage is low, parachute is non existent, your system may start failing.
[java] java.lang.OutOfMemoryError: Java heap space
[java] AsyncLogger error handling event seq=0, value='null':
[java] java.lang.OutOfMemoryError: Java heap space
[java] 06 Sep 2020 05:34:34,682 [ERROR]  (http-nio-0.0.0.0-8243-ClientPoller-1) org.apache.tomcat.util.net.NioEndpoint:
[java] java.lang.OutOfMemoryError: Java heap space
[java]  at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1043) ~[?:?]
[java]  at java.lang.Thread.run(Thread.java:834) [?:?]

ReaderDao.xml

<select id="downloadelements" resultMap="elementMap">
    select *
    from sample_table
    where element_timestamp between #{startTime} and #{endTime}
    and status=#{status}
</select>

ReaderDao.java

public interface ReaderDao {
Cursor<Element> downloadElements(final @NotNull @Param("startTime") ZonedDateTime startTime,
                                                final @NotNull @Param("endTime") ZonedDateTime endTime,
                                                final @NotNull @Param("status") String status);
}

数据库：Aurora PostgreSQL

Cursor<Element> elementList = downloadElements(start, end, status);
Iterator<Element> elementIterator = elementList.iterator();
while (elementIterator.hasNext()) {
    Element element = ElementIterator.next();
    log.info(element.toString());
}

<resultMap id="elementMap" type="Element">
    <constructor>
        <arg javaType="String" column="status"/>
        <arg javaType="java.time.ZonedDateTime" column="start_time"/>
        <arg javaType="java.util.Optional" jdbcType="TIMESTAMP"  column="end_time"/>
    </constructor>
</resultMap>

Answer 1

自 PostgreSQL driver does not support streaming results (as pointed out by @ave) you most likely want to implement result pagination. This is usually done either by using ORDER BY element_timestamp LIMIT <n> OFFSET <k> or by using keyset pagination。这些方法中的任何一种都会将较小批次的结果加载到内存中，希望通过允许 GC 分别释放每个批次来避免 OOME。

不过需要注意的是，有时增加可用堆内存是最简单的解决方案。 RAM 通常比程序员的时间更便宜，所以也许您应该首先估计您尝试处理的记录数以及需要多少内存。

从mybatis读取数据时OOM

OOM when reading data from mybatis

java

mybatis