休眠 "APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!"
Hibernate "APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!"
我最近在日志中收到以下消息，然后是 JVM 崩溃（由于 OOME）。我不知道该怎么做，非常感谢任何指导。
2015-03-19 21:15:02,457 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:26:29,543 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19397937
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c7d3838
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7aea62dd
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55622ff2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@74004a8
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
java.net.SocketOutputStream.write(SocketOutputStream.java:153)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:718)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
2015-03-19 21:56:59,137 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
at org.exist.storage.btree.BTree$BTreeNode.access(BTree.java:1239)
at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.access(BTree.java:1734)
at org.exist.storage.btree.BTree.query(BTree.java:379)
at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3a9d08ca
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ecdd11
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@44ff846d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5ce5850a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@eec1d04
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b8d4d9d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@53e9706d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@23d472cf
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4dbe4f8c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4c5e0203
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@54ac79fd
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@546e2bad
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@6b13cc83
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@57e185f8
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@60357d68
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@45231180
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@3021aa73
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@6bb437ca
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@2021c9e9
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@7d53637c
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@409c2c97
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@adc5929
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@241ca71a
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@42b26866
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@636b1c33
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@b160466
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4af34669
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b53e609
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062ebd4
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b6cfe8a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4b7c2380
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f9be748
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78108924
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@474b002
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ebee32f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e0fe017
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42aa175b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@637f5bac
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a017b77
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b4f2b78
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
com.mysql.jdbc.DatabaseMetaData.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
我们正在使用具有以下 c3p0 配置的 hiberanate 和 c3p0：
我们正在为休眠和 c3p0 使用以下 Maven 工件：
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.6.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.3.6.Final</version>
</dependency>
使用以下 c3p0 配置：
configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
.setProperty("hibernate.c3p0.idle_test_period", "1000")
.setProperty("hibernate.c3p0.min_size", "20")
.setProperty("hibernate.c3p0.max_size", "50")
.setProperty("hibernate.c3p0.timeout", "1800")
.setProperty("hibernate.c3p0.max_statements", "50")
服务器负载很轻~每秒 5 个查询。 (Java 8)
所以，直接的问题是连接池试图获取新的连接，但是尝试获取这些任务的任务冻结了很长一段时间，以至于 c3p0 决定任务必须死锁然后丢弃并替换了线程池。后来Thread pool挂在idle connection test tasks上
通常，"hung" 任务看起来像上面第一个 "Pool thread stack traces:" 标签下面的第二个线程：执行无法完成的网络 IO。您的情况很奇怪，因为三个线程中有两个没有卡在 IO 中。他们几乎没有开始做任何事情，但他们还没有活着。然后您遇到 OutOfMemoryError，并且您在空闲连接测试任务上遇到另一个明显的死锁，这些任务看起来也应该是实时的。
也许您的应用程序非常接近某种资源限制，导致 运行 非常缓慢？直截了当地说，您可以增加此应用程序可用的内存量（或修改它以减少内存占用）。线程池第二次尝试刷新并重新创建挂起的线程时，您会遇到 OOME，这不是由它直接引起的，但很可能是由不断增长的线程占用空间引起的。 （在您的日志中，之前是否有很多这些明显的死锁？如果您强制执行 JVM 线程转储，您是否看到周围有很多 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread 个实例，仍然挂在旧的，没有完成任务？
一些一般性评论：如果您的负载不超过 5 个并发查询，为什么池这么重？为什么不 min_size 5 岁左右，最多 10 岁左右？您的 max_statements 设置对于池中允许的连接数来说太小了。在你让事情顺利进行之前，我会完全忽略这一点。然后，为了获得更好的性能，您可以根据需要设置 more-straightforward-to-reason-about maxStatementsPerConnection 参数。
大多数情况下，您需要使应用程序的占用空间（内存？线程？）远低于分配给它的资源，方法是增加资源、减少占用空间或解决可能导致其资源占用空间增加的任何问题进入限制。我首先要缩小池，增大可用内存，然后将池配置得小得多。
我遇到了同样的异常，原因是数据库用户的密码错误...
c3p0 你是个有趣的人
我也是身份验证问题。我只是将域部分添加到数据库服务器名称中，一切正常。非常具有误导性的错误。
我最近在日志中收到以下消息，然后是 JVM 崩溃（由于 OOME）。我不知道该怎么做，非常感谢任何指导。
2015-03-19 21:15:02,457 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:26:29,543 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15da1b6b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b35b08a (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@51cfdd17 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19397937
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@5c7d3838
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7aea62dd
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@55622ff2
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@74004a8
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
java.net.SocketOutputStream.write(SocketOutputStream.java:153)
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3227)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1917)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2060)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1751)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3425)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2196)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:718)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.GeneratedConstructorAccessor306.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:302)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:282)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:560)
2015-03-19 21:56:59,137 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:608) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2015-03-19 21:56:59,143 [eXistThread-18676] ERROR (XQueryServlet.java [process]:566) - Java heap space
java.lang.OutOfMemoryError: Java heap space
at org.exist.storage.btree.BTree$BTreeNode.read(BTree.java:1269)
at org.exist.storage.btree.BTree$BTreeNode.access(BTree.java:1239)
at org.exist.storage.btree.BTree.getBTreeNode(BTree.java:460)
at org.exist.storage.btree.BTree.scanSequential(BTree.java:413)
at org.exist.storage.btree.BTree$BTreeNode.scanNextPage(BTree.java:2039)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1835)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.query(BTree.java:1759)
at org.exist.storage.btree.BTree$BTreeNode.access(BTree.java:1734)
at org.exist.storage.btree.BTree.query(BTree.java:379)
at org.exist.storage.structural.NativeStructuralIndexWorker.scanByType(NativeStructuralIndexWorker.java:259)
at org.exist.dom.VirtualNodeSet.getNodesFromIndex(VirtualNodeSet.java:457)
at org.exist.dom.VirtualNodeSet.realize(VirtualNodeSet.java:585)
at org.exist.dom.VirtualNodeSet.iterator(VirtualNodeSet.java:860)
at org.exist.dom.AbstractNodeSet.iterator(AbstractNodeSet.java:1)
at org.exist.storage.structural.NativeStructuralIndexWorker.findDescendantsByTagName(NativeStructuralIndexWorker.java:162)
at org.exist.xquery.LocationStep.getAttributes(LocationStep.java:645)
at org.exist.xquery.LocationStep.eval(LocationStep.java:434)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.Predicate.selectByNodeSet(Predicate.java:446)
at org.exist.xquery.Predicate.evalPredicate(Predicate.java:326)
at org.exist.xquery.LocationStep.processPredicate(LocationStep.java:251)
at org.exist.xquery.LocationStep.applyPredicate(LocationStep.java:238)
at org.exist.xquery.LocationStep.eval(LocationStep.java:462)
at org.exist.xquery.AbstractExpression.eval(AbstractExpression.java:71)
at org.exist.xquery.PathExpr.eval(PathExpr.java:264)
at org.exist.xquery.LetExpr.eval(LetExpr.java:142)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.LetExpr.eval(LetExpr.java:187)
at org.exist.xquery.BindingExpression.eval(BindingExpression.java:164)
2015-03-19 21:56:59,147 [Timer-0] WARN (ThreadPoolAsynchronousRunner.java [run]:624) - com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@6824f21c -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79180a12 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@243c6d0c (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@50191373 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3a9d08ca
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3ecdd11
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@44ff846d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5ce5850a
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@eec1d04
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b8d4d9d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@53e9706d
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@23d472cf
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4dbe4f8c
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@4c5e0203
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@54ac79fd
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@546e2bad
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@6b13cc83
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@57e185f8
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@60357d68
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@45231180
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@3021aa73
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@6bb437ca
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@2021c9e9
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@7d53637c
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@409c2c97
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@adc5929
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@241ca71a
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@42b26866
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@636b1c33
com.mchange.v2.resourcepool.BasicResourcePoolDestroyResourceTask@b160466
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4af34669
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b53e609
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062ebd4
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b6cfe8a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4b7c2380
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@4f9be748
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78108924
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@474b002
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2ebee32f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3e0fe017
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42aa175b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@637f5bac
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3a017b77
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7b4f2b78
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:408)
com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:370)
com.mysql.jdbc.MysqlIO.buildResultSetWithRows(MysqlIO.java:2532)
com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:477)
com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510)
com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2596)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1383)
com.mysql.jdbc.DatabaseMetaData.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1012)
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
com.newrelic.agent.TransactionService.addTransaction(TransactionService.java:142)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1104)
com.newrelic.agent.Transaction.getTransaction(Transaction.java:1087)
com.newrelic.agent.TracerService$TracerServiceImpl.getTracer(TracerService.java:136)
com.newrelic.agent.TracerService.getTracer(TracerService.java:41)
com.newrelic.agent.instrumentation.InvocationPoint.invoke(InvocationPoint.java:55)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java)
com.mysql.jdbc.DatabaseMetaData.forEach(DatabaseMetaData.java:4826)
com.mysql.jdbc.IterateBlock.doForAll(IterateBlock.java:50)
com.mysql.jdbc.DatabaseMetaData.getTables(DatabaseMetaData.java:4804)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnectionNoQuery(DefaultConnectionTester.java:185)
com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:62)
com.mchange.v2.c3p0.AbstractConnectionTester.activeCheckConnection(AbstractConnectionTester.java:67)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.testPooledConnection(C3P0PooledConnectionPool.java:368)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.refurbishIdleResource(C3P0PooledConnectionPool.java:310)
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask.run(BasicResourcePool.java:1999)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
我们正在使用具有以下 c3p0 配置的 hiberanate 和 c3p0：
我们正在为休眠和 c3p0 使用以下 Maven 工件：
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-core</artifactId>
<version>4.3.6.Final</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-c3p0</artifactId>
<version>4.3.6.Final</version>
</dependency>
使用以下 c3p0 配置：
configuration = new Configuration().setProperty("hibernate.dialect", "org.hibernate.dialect.MySQL5InnoDBDialect")
.setProperty("hibernate.connection.provider_class", "org.hibernate.connection.C3P0ConnectionProvider")
.setProperty("hibernate.c3p0.idle_test_period", "1000")
.setProperty("hibernate.c3p0.min_size", "20")
.setProperty("hibernate.c3p0.max_size", "50")
.setProperty("hibernate.c3p0.timeout", "1800")
.setProperty("hibernate.c3p0.max_statements", "50")
服务器负载很轻~每秒 5 个查询。 (Java 8)
所以，直接的问题是连接池试图获取新的连接，但是尝试获取这些任务的任务冻结了很长一段时间，以至于 c3p0 决定任务必须死锁然后丢弃并替换了线程池。后来Thread pool挂在idle connection test tasks上
通常，"hung" 任务看起来像上面第一个 "Pool thread stack traces:" 标签下面的第二个线程：执行无法完成的网络 IO。您的情况很奇怪，因为三个线程中有两个没有卡在 IO 中。他们几乎没有开始做任何事情，但他们还没有活着。然后您遇到 OutOfMemoryError，并且您在空闲连接测试任务上遇到另一个明显的死锁，这些任务看起来也应该是实时的。
也许您的应用程序非常接近某种资源限制，导致 运行 非常缓慢？直截了当地说，您可以增加此应用程序可用的内存量（或修改它以减少内存占用）。线程池第二次尝试刷新并重新创建挂起的线程时，您会遇到 OOME，这不是由它直接引起的，但很可能是由不断增长的线程占用空间引起的。 （在您的日志中，之前是否有很多这些明显的死锁？如果您强制执行 JVM 线程转储，您是否看到周围有很多 com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread 个实例，仍然挂在旧的，没有完成任务？
一些一般性评论：如果您的负载不超过 5 个并发查询，为什么池这么重？为什么不 min_size 5 岁左右，最多 10 岁左右？您的 max_statements 设置对于池中允许的连接数来说太小了。在你让事情顺利进行之前，我会完全忽略这一点。然后，为了获得更好的性能，您可以根据需要设置 more-straightforward-to-reason-about maxStatementsPerConnection 参数。
大多数情况下，您需要使应用程序的占用空间（内存？线程？）远低于分配给它的资源，方法是增加资源、减少占用空间或解决可能导致其资源占用空间增加的任何问题进入限制。我首先要缩小池，增大可用内存，然后将池配置得小得多。
我遇到了同样的异常，原因是数据库用户的密码错误...
c3p0 你是个有趣的人
我也是身份验证问题。我只是将域部分添加到数据库服务器名称中，一切正常。非常具有误导性的错误。