Mybatis return 大结果 xml 配置 spring

Question

我需要将数据从 oracle 中的 table 转储到 elasticsearch（1 亿条记录），我的JVM内存限制是256M，我使用下面的代码和配置从oracle（mybatis + spring）获取数据：接口：

package com.fudy.mapper;
import java.util.List;
import com.fudy.domain.Person;
public interface PersonMapper {
    List<Person> selectAllPerson();
}

xml 配置：

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mapper PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN" "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.fudy.mapper.PersonMapper">
    <resultMap type="com.fudy.domain.Person" id="PersonAlias">
        <id column="ID" property="id" />
        <result column="NAME" property="name" />
    </resultMap>
    <select id="selectAllPerson" fetchSize="10000" resultMap="PersonAlias">
        SELECT * FROM person
    </select>
</mapper>

ApplicationContext.xml :

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="http://www.springframework.org/schema/p"
    xmlns:tx="http://www.springframework.org/schema/tx" xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/tx
        http://www.springframework.org/schema/tx/spring-tx-3.0.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context-3.0.xsd">
    <context:annotation-config />
    <tx:annotation-driven transaction-manager="transactionManager" />

    <context:property-placeholder location="classpath:db.properties"/>

    <bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
        <property name="URL" value="${jdbc.url}" />
        <property name="user" value="${jdbc.user}"/>
        <property name="password" value="${jdbc.password}"/>
        <property name="connectionCachingEnabled" value="true"/>
    </bean>



    <!-- define the SqlSessionFactory, notice that configLocation is not needed 
        when you use MapperFactoryBean -->
    <bean id="sqlSessionFactory" class="org.mybatis.spring.SqlSessionFactoryBean">
        <property name="dataSource" ref="dataSource" />

        <property name="configLocation" value="classpath:Configuration.xml" />  

    </bean>

    <!-- scan for mappers and let them be autowired -->
    <bean class="org.mybatis.spring.mapper.MapperScannerConfigurer">
        <property name="basePackage" value="com.fudy.mapper" />
        <!-- optional unless there are multiple session factories defined -->
        <property name="sqlSessionFactoryBeanName" value="sqlSessionFactory" />
    </bean>

    <bean id="transactionManager"
          class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
        <property name="dataSource" ref="dataSource" />
    </bean> 

</beans>

我的 junit 测试：

package com.fudy.mapper;

import java.util.List;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.fudy.domain.Person;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration({"/ApplicationContext.xml"})
public class PersonMapperTest {
    @Autowired
    private PersonMapper mapper;
    @Test
    public void testSelectAllPerson() {
        List<Person> list = mapper.selectAllPerson();
        for ( Person person : list) {
            System.out.println(person.getId());
            System.out.println(person.getName());
            //TODO insert into elasticsearch 
        }
    }
}

从junit测试可以看出，Mybatis会将结果的整个列表return，这会导致内存不足的问题。 google 之后，我发现 ResultHandler 可能会解决问题，但是我找不到工作演示。

Answer 1

有两种选择：

使用ResultHandler
因为 3.4.1 使用 Cursor

结果处理程序

这就是您可以使用自定义的方式 ResultHandler:

PersonMapper.xml

<mapper namespace="com.fudy.mapper.PersonMapper">
  <resultMap type="com.fudy.domain.Person" id="PersonAlias">
    <id column="ID" property="id" />
    <result column="NAME" property="name" />
  </resultMap>
  <select id="selectAllPerson" resultMap="PersonAlias">
     SELECT * FROM person
  </select>
</mapper>

PersonMapper.java

public interface PersonMapper {
     void selectAllPersons(ResultHandler handler);
}

MyService.java

class PersonResultHandler implements ResultHandler {
    @Override
    public void handleResult(ResultContext context) { 
        Person person = (Person)context.getResultObject(); 
        // process person here
    }
};
PersonResultHandler handler = new PersonResultHandler();
PersonMapper personMapper = ...;
personMapper.selectAllPersons(handler);

光标

从mybatis 3.4.1开始可以return Cursor也就是Iterable可以这样使用（结果是有序的，见上文Cursor API java 文档获取详细信息):

PersonMapper.java

public interface PersonMapper {
     Cursor<Person> selectAllPersons();
}

MyService.java

PersonMapper personMapper = ...;
try (Cursor<Person> persons = personMapper.selectAllPersons()) {
   for (Person person : persons) {
      // process one person
   }
}

Answer 2

我看到 Roman Konoval 对你的问题给出了很好的答案，但我想补充一些东西。

我有一个类似的问题，我想遍历 table 而不是将它作为一个巨大的列表。我选择了 ResultHandler 解决方案，但我发现了一些麻烦。

即使您使用 ResultHandler，它也可能无法按预期工作。在您使用非常大的 table.

之前，我似乎可以正常工作

我发现有两件事很重要：

您放置交易的地方
设置一个 fetchSize 参数

直到我正确设置它，MyBatis 在运行 ResultHandler 之前消耗内存并获得整个 table。当我设置它时，MyBatis 不消耗内存并立即运行 ResultHandler。

我准备了四个案例:

交易设置不当。
交易已设置但未设置 fetchSize 参数。
设置了 fetchSize 参数但没有交易。
设置了事务和 fetchSize 参数。

前三种情况在我的计算机上导致 "java.lang.OutOfMemoryError: Java heap space"。最后一种情况立即运行 ResultHandler 并且不消耗内存。

我的示例使用 postgres 数据库并生成一些伪 table，但当然它可能很大 table。我示例中的 table 有 1_000_000 行。

MyEntity.java:

public class MyEntity {
    private Long id;

    private String text;

    public void setId(Long id) {
        this.id = id;
    }

    public void setText(String text) {
        this.text = text;
    }

    @Override
    public String toString() {
        return "MyEntity{" + "id=" + id + ", text='" + text + '\'' + '}';
    }
}

MyMapper.java:

@Mapper
@Component
public interface MyMapper {
    void selectAll1(ResultHandler handler);

    void selectAll2(ResultHandler handler);
}

MyResultHandler.java:

class MyResultHandler implements ResultHandler {
    @Override
    public void handleResult(ResultContext context) {
        System.out.println((MyEntity) context.getResultObject());
    }
}

resources/application.属性：

spring.datasource.url=jdbc:postgresql://localhost/temp1
spring.datasource.username=user1
spring.datasource.password=user1
mybatis.config-location=classpath:mybatis-config.xml

resources/mybatis-config.xml :

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration
        PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
    <mappers>
        <mapper resource="MyMapper.xml"/>
    </mappers>
</configuration>

resources/MyMapper.xml :

<!DOCTYPE mapper
        PUBLIC "-//mybatis.org//DTD Mapper 3.0//EN"
        "http://mybatis.org/dtd/mybatis-3-mapper.dtd">
<mapper namespace="com.example.mybatishugeselect.MyMapper">

    <select id="selectAll1" resultType="com.example.mybatishugeselect.MyEntity">
    SELECT generate_series(1,10000000) as id, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' as text
    </select>

    <select id="selectAll2" fetchSize="1000" resultType="com.example.mybatishugeselect.MyEntity">
    SELECT generate_series(1,10000000) as id, 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' as text
    </select>

</mapper>

MybatisHugeSelectApplicationTests.java :

@SpringBootTest
class MybatisHugeSelectApplicationTests {

    /* no transaction , no fetchSize parameter */
    @Test
    void case1(@Autowired MyMapper mapper) {
        mapper.selectAll1(new MyResultHandler());
    }

    /* no fetchSize parameter */
    @Test
    @Transactional
    void case2(@Autowired MyMapper mapper) {
        mapper.selectAll1(new MyResultHandler());
    }

    /* no transaction */
    @Test
    void case3(@Autowired MyMapper mapper) {
        mapper.selectAll2(new MyResultHandler());
    }

    /* good */
    @Test
    @Transactional
    void case4(@Autowired MyMapper mapper) {
        mapper.selectAll2(new MyResultHandler());
    }
}

只有最后一个 ( case4 ) 工作正常。

Mybatis return 大结果 xml 配置 spring

Mybatis return large result with xml configuration in spring

java

spring

mybatis

结果处理程序

光标