JDBC一次性数据转换批量更新

Question

我遇到一个情况，需要对近50万条记录进行数据转换。下面是过程

From Java we will call the DB and get a set of records (Older data)

These records will be passed to a webservice as input

The webservice response will be updated into the DB (correct data)

我对此做了一些研究，发现 SQL 批处理是一个不错的选择。

我打算做的是

Add 10,000 records to a batch (preparedstmnt. addBatch ();) via for loop

Then do a commit ( con.commit();)

preparedstmnt.clearBatch()

And then go back to step one until entire records are completed

尊敬的专家，您对此有何看法？approach.If您有更好的想法请也向我提出建议。此外，如果您在从 java 进行批量更新时有一些建议要记住，请也让我知道

还有 executeBatch() 最后会调用 clearBatch() 吗？或者我们是否需要在开始下一批之前在每批结束时显式调用 clearBatch？

Answer 1

基本上，如果您使用 Spring 批处理，它看起来像这样。正如我所说，其中一个优点是："It's no brainer"。易于实施和 Spring 驱动，这意味着如果它也是 Spring 驱动的，它将非常适合您当前的架构。这是大约。你的案例会是什么样子：

@Configuration
@EnableBatchProcessing
public class DBBatchProcess {

  @Bean
  public DataSource dataSource() {
        BasicDataSource dataSource = new BasicDataSource();
        dataSource.setDriverClassName("com.mysql.jdbc.Driver");
        dataSource.setUrl("jdbc:mysql://localhost:3306/myDB");
        dataSource.setUsername("username");
        dataSource.setPassword("password");
        return dataSource;
  }

  @Bean
  public ItemReader<InputEntity> reader(DataSource dataSource) {
    // Use some database reader. Like JdbcCursorItemReader, JdbcPagingItemReader etc.
    return new MyDatabaseReader();
  }

  @Bean
  public ItemProcessor<InputEntity, OutputEntity> processor() {
    // Do your conversion. Convert values from incoming entity to outcoming entity.
    return new MyProcessor();
  }

  @Bean
  public ItemWriter<OutputEntity> writer() {
    // Receive outcoming entity from processor and write it to database. You can use JdbcBatchItemWriter for instance.
    return new MyDatabaseWriter();
  }

  // Create a step. Provide reader, processor and writer. Determine chunk size.
  @Bean
  public Step step(StepBuilderFactory stepBuilderFactory, ItemReader<InputEntity> reader, ItemWriter<OutputEntity> writer,
            ItemProcessor<InputEntity, OutputEntity> processor) {

    return stepBuilderFactory.get("step")
                             .<InputEntity, OutputEntity>chunk(10000)
                             .reader(reader)
                             .processor(processor)
                             .writer(writer)
                             .build();
  }

  // Finally, create a job
  @Bean
  public Job job(JobBuilderFactory jobBuilderFactory, Step step) {
      return jobBuilderFactory.get("job")
                       .start(step)
                       .build();
  }

}

万一失败，您可以决定要做什么。您可以将侦听器附加到步骤或作业等。

JDBC一次性数据转换批量更新

JDBC Batch update for One time data Conversion

java

performance

jdbc

data-conversion

batch-processing