节点 CPU usage/load 尖峰
Node CPU usage/load spiking
我有许多 运行 JBoss 6 个节点，上面托管了一个应用程序。最近，我看到 most/all 他们一直在抱怨高 CPU 使用率和负载，这导致应用程序变慢。
所以昨晚当问题开始重新出现时，我进行了 JAVA 线程转储并在 nohup.out 文件中看到以下内容。
"http-/0.0.0.0:8091-Poller" daemon prio=10 tid=0x00007f4cb0171800 nid=0x2a1d in Object.wait() [0x00007f4d01dc0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007426621b0> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:732)
- locked <0x00000007426621b0> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at java.lang.Thread.run(Thread.java:745)
"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=10 tid=0x00007f4cc0056000 nid=0x2a18 waiting on condition [0x00007f4d022c5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1563)
at java.lang.Thread.run(Thread.java:745)
"ConnectionValidator" daemon prio=10 tid=0x00007f4cb4051800 nid=0x2a12 waiting on condition [0x00007f4d023c6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000740c29b68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at org.jboss.jca.core.connectionmanager.pool.validator.ConnectionValidator$ConnectionValidatorRunner.run(ConnectionValidator.java:263)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-/0.0.0.0:8091-8" daemon prio=10 tid=0x00007f4ce8046000 nid=0x2a2e in Object.wait() [0x00007f4d0055c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.pollResponse(BasicInvocationFuture.java:265)
- locked <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:216)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:404)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:223)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)
at com.ericsson.cac.sprint.adapters.hazelcast.HazelHelper.getObjectFromMap(HazelHelper.java:97)
at com.ericsson.cac.sprint.adapters.hazelcast.CacheAOP.testAop(CacheAOP.java:415)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
at com.ericsson.cac.sprint.adapters.QueryUsageProxyService$$EnhancerBySpringCGLIB$767524.queryPrepaidBalanceAndThresholdInfo(<generated>)
at com.ericsson.cac.sprint.selfcare.workflow.throttling.ThrottlingWorkflowImpl.getThrottleInfo(ThrottlingWorkflowImpl.java:127)
at com.ericsson.sprint.msdp.selfcare.controllers.abstracts.AbstractMyAccountController.usage(AbstractMyAccountController.java:529)
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:781)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:721)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
"hz._hzInstance_1_prod.event-5" prio=10 tid=0x00007f4cc8707000 nid=0x2abb waiting on condition [0x00007f4c85fde000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074f0b9ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:173)
"hz._hzInstance_1_prod.event-4" prio=10 tid=0x00007f4cc8704800 nid=0x2aba waiting on condition [0x00007f4c860df000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074f0ba1e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:173)
还有很多像这样的错误。
我还附上了完整的 Jboss 转储
知道为什么会这样吗？
这需要深入调查才能找出问题的根本原因，没有直接的方法可以告诉您问题出在哪里。
但是这里有一些指导你可以完成你的调查：
在线程转储中显示这个
"http-/0.0.0.0:8091-8" daemon prio=10 tid=0x00007f4ce8046000 nid=0x2a2e in Object.wait() [0x00007f4d0055c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.pollResponse(BasicInvocationFuture.java:265)
- locked <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:216)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:404)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:223)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)
at com.ericsson.cac.sprint.adapters.hazelcast.HazelHelper.getObjectFromMap(HazelHelper.java:97)
堆栈告诉线程正在等待对象监视器，这意味着另一个线程正在同步上下文中使用此对象，您需要跟踪锁定该对象的线程并查看当前正在做什么，在那里你可以找到是什么让那个线程持有对象太久了。
您可以使用 TDA 和 Thread Dump Analyzer 来完成。
您可能还需要检查内存状态。
我有许多 运行 JBoss 6 个节点，上面托管了一个应用程序。最近，我看到 most/all 他们一直在抱怨高 CPU 使用率和负载，这导致应用程序变慢。
所以昨晚当问题开始重新出现时，我进行了 JAVA 线程转储并在 nohup.out 文件中看到以下内容。
"http-/0.0.0.0:8091-Poller" daemon prio=10 tid=0x00007f4cb0171800 nid=0x2a1d in Object.wait() [0x00007f4d01dc0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007426621b0> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at org.apache.tomcat.util.net.JIoEndpoint$Poller.run(JIoEndpoint.java:732)
- locked <0x00000007426621b0> (a org.apache.tomcat.util.net.JIoEndpoint$Poller)
at java.lang.Thread.run(Thread.java:745)
"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=10 tid=0x00007f4cc0056000 nid=0x2a18 waiting on condition [0x00007f4d022c5000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1563)
at java.lang.Thread.run(Thread.java:745)
"ConnectionValidator" daemon prio=10 tid=0x00007f4cb4051800 nid=0x2a12 waiting on condition [0x00007f4d023c6000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000740c29b68> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at org.jboss.jca.core.connectionmanager.pool.validator.ConnectionValidator$ConnectionValidatorRunner.run(ConnectionValidator.java:263)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"http-/0.0.0.0:8091-8" daemon prio=10 tid=0x00007f4ce8046000 nid=0x2a2e in Object.wait() [0x00007f4d0055c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.pollResponse(BasicInvocationFuture.java:265)
- locked <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:216)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:404)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:223)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)
at com.ericsson.cac.sprint.adapters.hazelcast.HazelHelper.getObjectFromMap(HazelHelper.java:97)
at com.ericsson.cac.sprint.adapters.hazelcast.CacheAOP.testAop(CacheAOP.java:415)
at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
at com.ericsson.cac.sprint.adapters.QueryUsageProxyService$$EnhancerBySpringCGLIB$767524.queryPrepaidBalanceAndThresholdInfo(<generated>)
at com.ericsson.cac.sprint.selfcare.workflow.throttling.ThrottlingWorkflowImpl.getThrottleInfo(ThrottlingWorkflowImpl.java:127)
at com.ericsson.sprint.msdp.selfcare.controllers.abstracts.AbstractMyAccountController.usage(AbstractMyAccountController.java:529)
at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:781)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:721)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
"hz._hzInstance_1_prod.event-5" prio=10 tid=0x00007f4cc8707000 nid=0x2abb waiting on condition [0x00007f4c85fde000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074f0b9ef0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:173)
"hz._hzInstance_1_prod.event-4" prio=10 tid=0x00007f4cc8704800 nid=0x2aba waiting on condition [0x00007f4c860df000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074f0ba1e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at com.hazelcast.util.executor.StripedExecutor$Worker.run(StripedExecutor.java:173)
还有很多像这样的错误。
我还附上了完整的 Jboss 转储
知道为什么会这样吗？
这需要深入调查才能找出问题的根本原因，没有直接的方法可以告诉您问题出在哪里。
但是这里有一些指导你可以完成你的调查：
在线程转储中显示这个
"http-/0.0.0.0:8091-8" daemon prio=10 tid=0x00007f4ce8046000 nid=0x2a2e in Object.wait() [0x00007f4d0055c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.pollResponse(BasicInvocationFuture.java:265)
- locked <0x00000007c1965448> (a com.hazelcast.spi.impl.BasicInvocationFuture)
at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:216)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:193)
at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:404)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:223)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:84)
at com.ericsson.cac.sprint.adapters.hazelcast.HazelHelper.getObjectFromMap(HazelHelper.java:97)
堆栈告诉线程正在等待对象监视器，这意味着另一个线程正在同步上下文中使用此对象，您需要跟踪锁定该对象的线程并查看当前正在做什么，在那里你可以找到是什么让那个线程持有对象太久了。
您可以使用 TDA 和 Thread Dump Analyzer 来完成。
您可能还需要检查内存状态。