回滚失败:Spring-batch 数据库重启后连接关闭

Rollback fail : connection closed after database restart on Spring-batch

我在 Spring 批处理期间通过手动终止并重新启动我的 PostGreSQL 数据库来重现连接问题。我一直在使用DriverManagerDataSource,此时它刚刚崩溃,所以我不得不改变。现在我正在尝试使用 C3P0(与 Hibernate)。

post 很长,但主要是您不一定需要阅读的配置文件或堆栈跟踪。

我的主要问题是(我认为):c3p0 关闭了连接,然后 Sping-JTA 尝试回滚它,但失败了,这会终止该批处理。而且我不知道如何预防。


配置:

我正在使用 Spring、Spring 批处理 (4.2.3)、带 c3p0 (0.9.5.2) 和 postgresql 的休眠 (4.3.6)。

应用程序上下文:

<bean id="jpaVendorAdapter"
    class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
    <property name="database" value="POSTGRESQL" />
    <property name="databasePlatform" value="org.hibernate.dialect.PostgreSQLDialect" />
    <property name="generateDdl" value="false" />
    <property name="showSql" value="false" />
</bean>

<bean id="entityManagerFactory"
    class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="persistenceXmlLocation" value="classpath:META-INF/persistence.xml" />
    <property name="persistenceUnitName" value="ASO_PU" />
    <property name="jpaVendorAdapter" ref="jpaVendorAdapter" />
</bean>

<tx:annotation-driven transaction-manager="transactionManager" />
<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
    <property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>

persitence.xml

<persistence-unit name="ASO_PU" transaction-type="RESOURCE_LOCAL">
    <provider>org.hibernate.ejb.HibernatePersistence</provider>
    <class> ** my entities </class>
    <exclude-unlisted-classes>true</exclude-unlisted-classes>
    <properties>
        <property name="hibernate.dialect" value="org.hibernate.dialect.PostgreSQLDialect"/>

        <property name="hibernate.connection.driver_class" value="org.postgresql.Driver" />
        <property name="hibernate.connection.username" value="postgres" />
        <property name="hibernate.connection.password" value="postgres" />
        <property name="hibernate.connection.url" value="jdbc:postgresql://localhost:5434/aso_test" />
        <property name="hibernate.connection.provider_class" value="org.hibernate.connection.C3P0ConnectionProvider"/>

        <property name="hibernate.c3p0.maxSize" value="6"/> <!-- maxPoolSize -->
        <property name="hibernate.c3p0.minSize" value="2"/> <!-- minPoolSize -->
        <property name="hibernate.c3p0.maxStatements" value="30"/> <!-- maxStatements -->
        <property name="hibernate.c3p0.validate" value="true"/> <!-- testConnectionOnCheckout -->
        <property name="hibernate.c3p0.preferredTestQuery" value="SELECT 1"/> <!--  -->

            <property name="hibernate.c3p0.acquireRetryAttempts" value="121"/>
            <property name="hibernate.c3p0.acquireRetryDelay" value="2002"/>

        </properties>
    </persistence-unit>

会发生什么:

当应用程序在我杀死它后尝试连接到数据库时,它会警告我,然后等到数据库再次启动(我不认为它是相关的,但无论如何这是跟踪:)

ATTENTION: [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
31 août 2016 11:44:54 com.mchange.v2.c3p0.impl.NewPooledConnection handleThrowable
ATTENTION: [c3p0] Another error has occurred [ org.postgresql.util.PSQLException: This connection has been closed. ] which will not be reported to listeners!
org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:837)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.rollback(AbstractJdbc2Connection.java:854)
    at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:860)
    at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doRollback(JdbcTransaction.java:163)
    at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.rollback(AbstractTransactionImpl.java:211)
    at org.hibernate.jpa.internal.TransactionImpl.rollback(TransactionImpl.java:108)
    at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:544)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:853)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:830)
    at org.springframework.transaction.support.TransactionTemplate.rollbackOnException(TransactionTemplate.java:164)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doInChunkContext(TaskletStep.java:271)
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77)
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368)
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198)
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:165)
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:135)
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128)
    at fr.aso.batch.test.MyBatchTest.test(MyBatchTest.java:100)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.junit.runners.model.FrameworkMethod.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access[=13=]0(ParentRunner.java:58)
    at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
31 août 2016 11:44:54 com.mchange.v2.c3p0.impl.NewPooledConnection handleThrowable
ATTENTION: [c3p0] A PooledConnection that has already signalled a Connection error is still in use!

发生此异常后,批处理挂起。一旦数据库再次启动(1 分钟后),整个批次就会崩溃:

     01/09/2016 09:22:47.537 [ERROR]    REI - org.springframework.batch.core.job.AbstractJob            Exception encountered in afterStep callback 
org.springframework.transaction.TransactionSystemException: Could not roll back JPA transaction; nested exception is javax.persistence.PersistenceException: unexpected error when rollbacking
    at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:548) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:853) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:830) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:503) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:285) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at $Proxy204.afterJob(Unknown Source) ~[na:na]
    at org.springframework.batch.core.listener.CompositeJobExecutionListener.afterJob(CompositeJobExecutionListener.java:60) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:346) ~[spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.4.RELEASE.jar:3.0.4.RELEASE]
    at fr.aso.batch.OpsTest.testRecuperationEtabsEmployantSalariesCNAMJob(OpsTest.java:100) [test-classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_30]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_30]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_30]
    at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_30]
    at org.junit.runners.model.FrameworkMethod.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12]
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12]
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12]
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:254) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.junit.runners.ParentRunner.run(ParentRunner.java:290) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.access[=14=]0(ParentRunner.java:58) [junit-4.12.jar:4.12]
    at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12]
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193) [spring-test-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49) [.cp/:na]
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) [.cp/:na]
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) [.cp/:na]
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) [.cp/:na]
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) [.cp/:na]
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) [.cp/:na]
Caused by: javax.persistence.PersistenceException: unexpected error when rollbacking
    at org.hibernate.jpa.internal.TransactionImpl.rollback(TransactionImpl.java:111) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    at org.springframework.orm.jpa.JpaTransactionManager.doRollback(JpaTransactionManager.java:544) ~[spring-orm-4.2.3.RELEASE.jar:4.2.3.RELEASE]
    ... 43 common frames omitted
Caused by: org.hibernate.TransactionException: rollback failed
    at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.rollback(AbstractTransactionImpl.java:217) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
    at org.hibernate.jpa.internal.TransactionImpl.rollback(TransactionImpl.java:108) ~[hibernate-entitymanager-4.3.5.Final.jar:4.3.5.Final]
    ... 44 common frames omitted
Caused by: org.hibernate.TransactionException: unable to rollback against JDBC connection
    at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doRollback(JdbcTransaction.java:167) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
    at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.rollback(AbstractTransactionImpl.java:211) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
    ... 45 common frames omitted
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
    at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:837) ~[postgresql-9.3-1101.jdbc4.jar:na]
    at org.postgresql.jdbc2.AbstractJdbc2Connection.rollback(AbstractJdbc2Connection.java:854) ~[postgresql-9.3-1101.jdbc4.jar:na]
    at com.mchange.v2.c3p0.impl.NewProxyConnection.rollback(NewProxyConnection.java:860) ~[c3p0-0.9.2.1.jar:0.9.2.1]
    at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doRollback(JdbcTransaction.java:163) ~[hibernate-core-4.3.6.Final.jar:4.3.6.Final]
    ... 46 common frames omitted

从这个堆栈跟踪来看,我认为它是这样的:当 DB 死亡时 c3p0 关闭连接,当 JTA 试图回滚时,c3p0 挂起直到 DB 再次活跃。一旦它再次启动,回滚就可以开始,但它试图在关闭的连接上回滚,因此异常会杀死整个批次。

我以为当连接关闭时,c3p0 会得到一个新的,但我想这在回滚时是不可能的(它会在?之后尝试),并且 Spring JTA 必须尝试在尝试重新连接之前回滚。


问题:

首先,我不知道我说的是否正确...JTA 是否可能尝试回滚已关闭的事务?

如果是这样,我没有发现任何关于如何处理 JTA 回滚期间的问题,或者可能是 c3p0?冬眠?甚至 Spring 批次 ?

我能做什么? (要么确定问题是什么,要么如何解决)。

感谢任何帮助。提前谢谢你(好长post ...)


备注

崩溃随时可能发生,这里它发生在处理器上是因为我杀死了数据库。但我也可能在 spring 批量更新其 job_instance_execution table 或任何时候发生,但我认为这一切都由 spring-JTA/hibernate/c3p0 管理,所以我正在看这个而不是spring-批次。

除非您使用的是正式的分布式事务(即 XAConnection),否则执行 JDBC 事务的级别是 Connection 对象。如果 Connection 因未提交的事务性工作而终止,则该工作将丢失,就是这样。对此无能为力。如果您希望在连接中断时提交部分工作,请以更细粒度的方式定义您的事务,以便立即提交您希望提交的工作。

除非你设置了c3p0.unreturnedConnectionTimeout(不是你的问题),c3p0 永远不会close() 从应用程序下连接出来。 c3p0 正在报告来自 Postgres 的错误,它注意到底层连接已关闭。在您的应用程序(直接或通过休眠)在连接上调用 close() 之前,无论它有多破烂,它都会结帐。

在您描述的所有内容中,唯一不寻常的是第二次投诉前的停顿。可能的顺序是:

  1. 连接终止
  2. 应用程序尝试使用它时出现异常
  3. 当应用程序试图将其回滚时发生进一步的异常(您显示的第一个堆栈跟踪)
  4. 希望 JPA/hibernate/whatever 小心地调用连接上的 close() 到 return 它来控制连接池
  5. 在继续之前,JPA/hibernate 等待新的有效连接可用。
  6. 然后它会向您的应用程序报告先前的失败并出现异常。

第 5 步和第 6 步的顺序有点令人惊讶。从概念上讲,JPA 没有理由不能立即向应用程序报告故障。除此之外,一切都是应该的。

"What can I do?"

确保您编写的应用程序能够理解 JPA 失败的可能性,并确保应用程序始终处于一致的状态,尽管如此。失败时,您的应用程序可能会重试失败的工作,或者它可能只是将失败报告给客户端(无论是通过某种消息的最终用户,还是通过异常报告给其他软件)。如果即使在重试之后失败仍然存在,最终您可能不得不报告失败(除非有某种方法可以在没有数据库的情况下解决并从失败中恢复,这是不太可能的)。应用程序无法确保网络上的数据库不会简单地消失。

在 c3p0 级别,如果您愿意,可以设置 testConnectionOnCheckinidleConnectionTestPeriod 以确保在数据库丢失频繁的情况下及时清除错误的连接。 (就正确性而言,您当前的设置,通过 hibernate.c3p0.validate 使用 testConnectionOnCheckout 很好,但它会导致池延迟替换连接,因为客户需要它们)。有关连接测试文档,请参阅 here and here