使用 Zipkin TracingStatementInterceptor 启动 c3po 连接池的死锁
Deadlock starting c3po connection pool with Zipkin TracingStatementInterceptor
尝试通过在 Spring 引导应用程序中包含以下库来添加 Zipkin mysql 跟踪工具
compile group: 'io.zipkin.brave', name: 'brave-instrumentation-mysql', version: '5.6.3'
并将以下内容附加到我的 jdbc 连接字符串中：
?statementInterceptors=brave.mysql.TracingStatementInterceptor
现有数据库连接正在使用配置如下的 c3po 连接池：
<bean id="dataSourceBean" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close" primary="true">
<property name="driverClass" value="${db.driver}" />
<property name="jdbcUrl" value="${db.url}" />
<property name="user" value="${db.username}" />
<property name="password" value="${db.password}" />
<property name="initialPoolSize" value="${db.initial.pool.size}" />
<property name="minPoolSize" value="${db.min.pool.size}" />
<property name="maxPoolSize" value="${db.max.pool.size}" />
<property name="idleConnectionTestPeriod" value="30" />
<property name="maxIdleTimeExcessConnections" value="120" />
</bean>
但 运行 在尝试从池中获取连接时启动服务时出现问题：
2019-05-09 22:54:40.752 WARN [api,,,] 31975 --- [-AdminTaskTimer] c.m.v.a.ThreadPoolAsynchronousRunner : com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@222a008d -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5cf72aaf
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@67c13f4f
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@dff1fcc
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@775717e4
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1290e9f3
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7cc597a9
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@9b1e682
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@46963867
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@53dc8e37
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3f4233a
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0,5,main]
org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
brave.Tracer.nextContext(Tracer.java:252)
brave.Tracer.newRootContext(Tracer.java:212)
brave.Tracer.newTrace(Tracer.java:142)
brave.Tracer.nextSpan(Tracer.java:436)
brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1,5,main]
org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
brave.Tracer.nextContext(Tracer.java:252)
brave.Tracer.newRootContext(Tracer.java:212)
brave.Tracer.newTrace(Tracer.java:142)
brave.Tracer.nextSpan(Tracer.java:436)
brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2,5,main]
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:204)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:392)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean(AbstractBeanFactory.java:356)
org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda0/35986993.getObject(Unknown Source)
org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:389)
org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
brave.Tracer.nextContext(Tracer.java:252)
brave.Tracer.newRootContext(Tracer.java:212)
brave.Tracer.newTrace(Tracer.java:142)
brave.Tracer.nextSpan(Tracer.java:436)
brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
看起来跟踪正在触发 old-fashioned inverted-lock-order 死锁冻结试图获取新连接的线程。
三个死锁线程中的最后一个是 trying to get a lock on some singleton or bean。它已经通过并可能获得了对 GenericScope.
的锁定
另外两个线程是trying to acquire a lock on a GenericScope，大概第一个线程有。
从 zipkin 代码意外重新进入 spring 正在产生死锁。 c3p0 有一个 fixed-size 线程池，当它的所有线程（这里只有 3 个，c3p0 的默认值）被持久冻结时，它会发出通知，然后（在这种情况下非常正确）声明一个死锁并且替换阻塞的线程以期恢复。
c3p0 恢复了吗？这是罕见的还是经常发生的僵局？要防止这种僵局，您可以轻松做的事情并不多，我认为您要么必须容忍它，要么不使用仪器。
尝试通过在 Spring 引导应用程序中包含以下库来添加 Zipkin mysql 跟踪工具
compile group: 'io.zipkin.brave', name: 'brave-instrumentation-mysql', version: '5.6.3'
并将以下内容附加到我的 jdbc 连接字符串中：
?statementInterceptors=brave.mysql.TracingStatementInterceptor
现有数据库连接正在使用配置如下的 c3po 连接池：
<bean id="dataSourceBean" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close" primary="true">
<property name="driverClass" value="${db.driver}" />
<property name="jdbcUrl" value="${db.url}" />
<property name="user" value="${db.username}" />
<property name="password" value="${db.password}" />
<property name="initialPoolSize" value="${db.initial.pool.size}" />
<property name="minPoolSize" value="${db.min.pool.size}" />
<property name="maxPoolSize" value="${db.max.pool.size}" />
<property name="idleConnectionTestPeriod" value="30" />
<property name="maxIdleTimeExcessConnections" value="120" />
</bean>
但 运行 在尝试从池中获取连接时启动服务时出现问题：
2019-05-09 22:54:40.752 WARN [api,,,] 31975 --- [-AdminTaskTimer] c.m.v.a.ThreadPoolAsynchronousRunner : com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@222a008d -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5cf72aaf
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@67c13f4f
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@dff1fcc
on thread: C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@775717e4
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1290e9f3
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@7cc597a9
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@9b1e682
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@46963867
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@53dc8e37
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3f4233a
Pool thread stack traces:
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#0,5,main]
org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
brave.Tracer.nextContext(Tracer.java:252)
brave.Tracer.newRootContext(Tracer.java:212)
brave.Tracer.newTrace(Tracer.java:142)
brave.Tracer.nextSpan(Tracer.java:436)
brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#1,5,main]
org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:388)
org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
brave.Tracer.nextContext(Tracer.java:252)
brave.Tracer.newRootContext(Tracer.java:212)
brave.Tracer.newTrace(Tracer.java:142)
brave.Tracer.nextSpan(Tracer.java:436)
brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
Thread[C3P0PooledConnectionPoolManager[identityToken->z8kflta218cyxuc1srbxxo|30f51f8a]-HelperThread-#2,5,main]
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:204)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:392)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1305)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1144)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean(AbstractBeanFactory.java:356)
org.springframework.beans.factory.support.AbstractBeanFactory$$Lambda0/35986993.getObject(Unknown Source)
org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:389)
org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:186)
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:353)
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35)
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)
brave.sampler.Sampler$$EnhancerBySpringCGLIB$$d8298fbe.isSampled(<generated>)
brave.Tracer.nextContext(Tracer.java:252)
brave.Tracer.newRootContext(Tracer.java:212)
brave.Tracer.newTrace(Tracer.java:142)
brave.Tracer.nextSpan(Tracer.java:436)
brave.propagation.ThreadLocalSpan.next(ThreadLocalSpan.java:119)
brave.mysql.TracingStatementInterceptor.preProcess(TracingStatementInterceptor.java:33)
com.mysql.jdbc.NoSubInterceptorWrapper.preProcess(NoSubInterceptorWrapper.java:71)
com.mysql.jdbc.MysqlIO.invokeStatementInterceptorsPre(MysqlIO.java:3007)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2671)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2828)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2777)
com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1651)
com.mysql.jdbc.ConnectionImpl.loadServerVariables(ConnectionImpl.java:4215)
com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3609)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2550)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2320)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolPooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
com.mchange.v2.resourcepool.BasicResourcePool.access0(BasicResourcePool.java:44)
com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
看起来跟踪正在触发 old-fashioned inverted-lock-order 死锁冻结试图获取新连接的线程。
三个死锁线程中的最后一个是 trying to get a lock on some singleton or bean。它已经通过并可能获得了对 GenericScope.
另外两个线程是trying to acquire a lock on a GenericScope，大概第一个线程有。
从 zipkin 代码意外重新进入 spring 正在产生死锁。 c3p0 有一个 fixed-size 线程池，当它的所有线程（这里只有 3 个，c3p0 的默认值）被持久冻结时，它会发出通知，然后（在这种情况下非常正确）声明一个死锁并且替换阻塞的线程以期恢复。
c3p0 恢复了吗？这是罕见的还是经常发生的僵局？要防止这种僵局，您可以轻松做的事情并不多，我认为您要么必须容忍它，要么不使用仪器。