Jenkins 在执行器上使用 100% CPU 构建作业
Jenkins build job using 100% CPU on executors
一段时间以来，我们看到我们的 Jenkins 机器被锁定在 100% CPU（或 200% 或 400%，取决于核心数量）根据 top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23376 ec2-user 20 0 4220m 100m 0 S 99.2 1.3 324945:21 java
这些 JVM 进程甚至在构建完成后仍然存在，而当前没有构建 运行。在主节点和从节点上构建都会出现此问题。 JVM 运行 从属代理本身具有完全正常的 CPU 用法。
一旦我终于能够获得线程转储，只有一个非系统线程可运行且未等待锁：
"main" #1 prio=5 os_prio=0 tid=0x00007f0834008800 nid=0x5b51 runnable [0x00007f083abf3000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Harmonized.detach(MethodGraph.java:878)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store$Entry$Resolved.asNode(MethodGraph.java:1331)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store.asGraph(MethodGraph.java:1138)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.compile(MethodGraph.java:507)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$AbstractBase.compile(MethodGraph.java:423)
at net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:489)
at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamqicTypeBuilder.make(SubclassDynamicTypeBuilder.java:153)
at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$Delegator.make(DynamicType.java:2508)
at org.mockito.internal.creation.bytebuddy.MockBytecodeGenerator.generateMockClass(MockBytecodeGenerator.java:60)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.generate(CachingMockBytecodeGenerator.java:72)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.getOrGenerateMockClass(CachingMockBytecodeGenerator.java:64)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator.get(CachingMockBytecodeGenerator.java:27)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createProxyClass(ByteBuddyMockMaker.java:54)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:27)
at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:32)
at org.mockito.internal.MockitoCore.mock(MockitoCore.java:55)
at org.mockito.Mockito.mock(Mockito.java:1449)
at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:33)
at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:16)
at org.mockito.internal.configuration.DefaultAnnotationEngine.createMockFor(DefaultAnnotationEngine.java:43)
at org.mockito.internal.configuration.DefaultAnnotationEngine.process(DefaultAnnotationEngine.java:66)
at org.mockito.internal.configuration.InjectingAnnotationEngine.processIndependentAnnotations(InjectingAnnotationEngine.java:71)
at org.mockito.internal.configuration.InjectingAnnotationEngine.process(InjectingAnnotationEngine.java:55)
at org.mockito.MockitoAnnotations.initMocks(MockitoAnnotations.java:108)
at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.withBefores(JUnit45AndHigherRunnerImpl.java:27)
at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:276)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access[=12=]0(ParentRunner.java:58)
at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
我进行了几次连续的线程转储，这个线程几乎总是在完全相同的位置 - HashMap.put(HashMap.java:611) - 而 JVM 占用了 100% CPU。发生这种情况时，我还没有机会附加调试器，但我只是想看看是否有人认为这是一个已知错误。对我来说，它看起来像是某种热旋转的无限循环，它试图一遍又一遍地将一些项目添加到哈希映射中。这可能与 Jenkins 有关，但在堆栈跟踪中出现的其他主要参与者显然是 Maven Surefire、JUnit、Mockito 和 ByteBuddy（抱歉包含所有这些标签）。
我无法可靠地重现这个问题，不幸的是，我也不知道我们数百个构建中的哪一个留下了这些搞砸了的 JVM。为了完整起见，环境是 Jenkins 2.46.1、Maven 3.3.3、Surefire 2.19.1、JUnit 4.12，但不幸的是一系列不同的 Mockito（以及 ByteBuddy）版本。我希望有人认识到这是其中一个相关组件中的已知错误，并可以提出解决方法...
你能用最新版本的 Mockito 重试吗？这似乎是我们用于模拟创建的新锁定机制的问题。以前，我们只对所有类型使用一个锁，现在锁的粒度更细了。
您看到的堆栈跟踪是在模拟创建中，其中 Byte Buddy 构建了由 class 定义的方法结构。它需要这样做，以在层次结构中解析桥接方法。这是一个相当昂贵的操作，但它不应该无休止地旋转。您有 class 层级非常深（超过 50 层）的实体吗？
一段时间以来，我们看到我们的 Jenkins 机器被锁定在 100% CPU（或 200% 或 400%，取决于核心数量）根据 top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23376 ec2-user 20 0 4220m 100m 0 S 99.2 1.3 324945:21 java
这些 JVM 进程甚至在构建完成后仍然存在，而当前没有构建 运行。在主节点和从节点上构建都会出现此问题。 JVM 运行 从属代理本身具有完全正常的 CPU 用法。
一旦我终于能够获得线程转储，只有一个非系统线程可运行且未等待锁：
"main" #1 prio=5 os_prio=0 tid=0x00007f0834008800 nid=0x5b51 runnable [0x00007f083abf3000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Harmonized.detach(MethodGraph.java:878)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store$Entry$Resolved.asNode(MethodGraph.java:1331)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store.asGraph(MethodGraph.java:1138)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.compile(MethodGraph.java:507)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$AbstractBase.compile(MethodGraph.java:423)
at net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:489)
at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamqicTypeBuilder.make(SubclassDynamicTypeBuilder.java:153)
at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$Delegator.make(DynamicType.java:2508)
at org.mockito.internal.creation.bytebuddy.MockBytecodeGenerator.generateMockClass(MockBytecodeGenerator.java:60)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.generate(CachingMockBytecodeGenerator.java:72)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.getOrGenerateMockClass(CachingMockBytecodeGenerator.java:64)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator.get(CachingMockBytecodeGenerator.java:27)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createProxyClass(ByteBuddyMockMaker.java:54)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:27)
at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:32)
at org.mockito.internal.MockitoCore.mock(MockitoCore.java:55)
at org.mockito.Mockito.mock(Mockito.java:1449)
at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:33)
at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:16)
at org.mockito.internal.configuration.DefaultAnnotationEngine.createMockFor(DefaultAnnotationEngine.java:43)
at org.mockito.internal.configuration.DefaultAnnotationEngine.process(DefaultAnnotationEngine.java:66)
at org.mockito.internal.configuration.InjectingAnnotationEngine.processIndependentAnnotations(InjectingAnnotationEngine.java:71)
at org.mockito.internal.configuration.InjectingAnnotationEngine.process(InjectingAnnotationEngine.java:55)
at org.mockito.MockitoAnnotations.initMocks(MockitoAnnotations.java:108)
at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.withBefores(JUnit45AndHigherRunnerImpl.java:27)
at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:276)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access[=12=]0(ParentRunner.java:58)
at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)
我进行了几次连续的线程转储，这个线程几乎总是在完全相同的位置 - HashMap.put(HashMap.java:611) - 而 JVM 占用了 100% CPU。发生这种情况时，我还没有机会附加调试器，但我只是想看看是否有人认为这是一个已知错误。对我来说，它看起来像是某种热旋转的无限循环，它试图一遍又一遍地将一些项目添加到哈希映射中。这可能与 Jenkins 有关，但在堆栈跟踪中出现的其他主要参与者显然是 Maven Surefire、JUnit、Mockito 和 ByteBuddy（抱歉包含所有这些标签）。
我无法可靠地重现这个问题，不幸的是，我也不知道我们数百个构建中的哪一个留下了这些搞砸了的 JVM。为了完整起见，环境是 Jenkins 2.46.1、Maven 3.3.3、Surefire 2.19.1、JUnit 4.12，但不幸的是一系列不同的 Mockito（以及 ByteBuddy）版本。我希望有人认识到这是其中一个相关组件中的已知错误，并可以提出解决方法...
你能用最新版本的 Mockito 重试吗？这似乎是我们用于模拟创建的新锁定机制的问题。以前，我们只对所有类型使用一个锁，现在锁的粒度更细了。
您看到的堆栈跟踪是在模拟创建中，其中 Byte Buddy 构建了由 class 定义的方法结构。它需要这样做，以在层次结构中解析桥接方法。这是一个相当昂贵的操作，但它不应该无休止地旋转。您有 class 层级非常深（超过 50 层）的实体吗？