并行执行测试的周期性问题

Question

下午好！我使用以下堆栈来实现自动化测试：Java 8、Maven、Jenkins 用于测试的自动化执行。有时（不是每次，大约占所有执行的 3-5%）我在并行执行期间遇到测试问题。并行执行由 Jenkins 和 jenkins 文件提供。詹金斯文件的示例结构：

stage('First suite'){
            parallel {
                stage('1 test'){
                }
                stage('2 test'){
                }
            }
}

上次出现以下错误：

[ERROR] java.lang.NullPointerException
[ERROR]     at java.util.Properties$LineReader.readLine(Properties.java:434)
[ERROR]     at java.util.Properties.load0(Properties.java:353)
[ERROR]     at java.util.Properties.load(Properties.java:341)
[ERROR]     at org.apache.maven.surefire.booter.SystemPropertyManager.loadProperties(SystemPropertyManager.java:50)
[ERROR]     at org.apache.maven.surefire.booter.BooterDeserializer.<init>(BooterDeserializer.java:62)
[ERROR]     at org.apache.maven.surefire.booter.ForkedBooter.setupBooter(ForkedBooter.java:109)
[ERROR]     at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:561)
[ERROR]     at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project webuicheck: There are test failures.

[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 1
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:748)
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:305)
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:265)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
[ERROR]     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR]     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR]     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR]     at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR]     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR]     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR]     at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR]     at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR]     at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR]     at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR]     at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]     at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR]     at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

我已尝试遵循有关以下方面的所有建议：

更新 maven-surefire-plugin（现在我有一个最新版本）；
在pom.xml中设置参数：

                    <systemPropertyVariables>
                        <xmlOutputDir>${project.build.directory}/surefire</xmlOutputDir>
                    </systemPropertyVariables>
                    <classesDirectory>${project.build.outputDirectory}</classesDirectory>
                    <useSystemClassLoader>false</useSystemClassLoader>
                    <useManifestOnlyJar>false</useManifestOnlyJar>
                    <parallel>classes</parallel>
                    <forkCount>10</forkCount>
                    <reuseForks>true</reuseForks>
                    <useUnlimitedThreads>true</useUnlimitedThreads>
                    <argLine>-Xmx1024m -Xms64m</argLine>

我很绝望，不知道还能做什么。也许您在 jenkins 的并行执行期间遇到过同样的问题（单个测试执行总是成功完成）。谢谢你，祝你有美好的一天！

Answer 1

您不应该运行 Maven 并行构建两次。他们会互相覆盖。这是非常糟糕的模式。

你能告诉我舞台里面有什么吗？

            stage('1 test'){
            }
            stage('2 test'){
            }

Answer 2

@tibor17，美好的一天！当然是：

stage('First suite'){
   steps {
       catchError(buildResult: 'FAILURE', stageResult: 'FAILURE'){
          sh 'mvn -e -Dtest=FirstSuite test'
       }
    }
}
stage('Second suite'){
   steps {
       catchError(buildResult: 'FAILURE', stageResult: 'FAILURE'){
          sh 'mvn -e -Dtest=SecondSuite test'
       }
    }
}

但是你是什么意思'overwrite each other'。最后，在执行我的 mvn 测试并且没有任何内容被覆盖的所有阶段结束时，我得到了完整的 Allure 报告。问题是：很少（我所有启动的 2-3%）这会导致错误，并且一个或多个 tests/stages 未执行并因 maven-surefire-plugin 错误而失败。

Answer 3

下午好！我生成了有效的 POM：

<?xml version="1.0" encoding="Cp1251"?>
<!-- ====================================================================== -->
<!--                                                                        -->
<!-- Generated by Maven Help Plugin on 2021-07-15T12:23:27+03:00            -->
<!-- See: http://maven.apache.org/plugins/maven-help-plugin/                -->
<!--                                                                        -->
<!-- ====================================================================== -->
<!-- ====================================================================== -->
<!--                                                                        -->
<!-- Effective POM for project 'webuicheck:jar:1.0-SNAPSHOT'      -->
<!--                                                                        -->
<!-- ====================================================================== -->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/
xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId></groupId>
  <artifactId>webuicheck</artifactId>
  <version>1.0-SNAPSHOT</version>
  <properties>
    <allure.version>2.13.9</allure.version>
    <aspectj.version>1.9.7.M3</aspectj.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.12</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.codeborne</groupId>
      <artifactId>selenide</artifactId>
      <version>5.22.3</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.opencsv</groupId>
      <artifactId>opencsv</artifactId>
      <version>5.4</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-selenide</artifactId>
      <version>2.13.9</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-junit4</artifactId>
      <version>2.13.9</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>javax.xml.bind</groupId>
      <artifactId>jaxb-api</artifactId>
      <version>2.3.0</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.assertj</groupId>
      <artifactId>assertj-core</artifactId>
      <version>3.19.0</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>com.tngtech.junit.dataprovider</groupId>
      <artifactId>junit4-dataprovider</artifactId>
      <version>2.6</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-nop</artifactId>
      <version>1.7.24</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.poi</groupId>
      <artifactId>poi-ooxml</artifactId>
      <version>4.1.2</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.oracle.database.jdbc</groupId>
      <artifactId>ojdbc8</artifactId>
      <version>19.8.0.0</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
  <repositories>
    <repository>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <id>central</id>
      <name>Central Repository</name>
      <url>https://repo.maven.apache.org/maven2</url>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <releases>
        <updatePolicy>never</updatePolicy>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <id>central</id>
      <name>Central Repository</name>
      <url>https://repo.maven.apache.org/maven2</url>
    </pluginRepository>
  </pluginRepositories>
  <build>
    <sourceDirectory>C:\Projects\webuicheck\src\main\java</sourceDirectory>
    <scriptSourceDirectory>C:\Projects\webuicheck\src\main\scripts</scriptSourceDirectory>
    <testSourceDirectory>C:\Projects\webuicheck\src\test\java</testSourceDirectory>
    <outputDirectory>C:\Projects\webuicheck\target\classes</outputDirectory>
    <testOutputDirectory>C:\Projects\webuicheck\target\test-classes</testOutputDirectory>
    <resources>
      <resource>
        <directory>C:\Projects\webuicheck\src\main\resources</directory>
      </resource>
    </resources>
    <testResources>
      <testResource>
        <filtering>true</filtering>
        <directory>C:\Projects\webuicheck\src\test\resources\properties</directory>
        <includes>
          <include>**/*.properties</include>
        </includes>
      </testResource>
    </testResources>
    <directory>C:\Projects\webuicheck\target</directory>
    <finalName>webuicheck-1.0-SNAPSHOT</finalName>
    <pluginManagement>
      <plugins>
        <plugin>
          <artifactId>maven-antrun-plugin</artifactId>
          <version>1.3</version>
        </plugin>
        <plugin>
          <artifactId>maven-assembly-plugin</artifactId>
          <version>2.2-beta-5</version>
        </plugin>
        <plugin>
          <artifactId>maven-dependency-plugin</artifactId>
          <version>2.8</version>
        </plugin>
        <plugin>
          <artifactId>maven-release-plugin</artifactId>
          <version>2.5.3</version>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <executions>
          <execution>
            <id>default-compile</id>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
            <configuration>
              <source>1.8</source>
              <target>1.8</target>
            </configuration>
          </execution>
          <execution>
            <id>default-testCompile</id>
            <phase>test-compile</phase>
            <goals>
              <goal>testCompile</goal>
            </goals>
            <configuration>
              <source>1.8</source>
              <target>1.8</target>
            </configuration>
          </execution>
        </executions>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-resources-plugin</artifactId>
        <version>3.0.2</version>
        <executions>
          <execution>
            <id>default-testResources</id>
            <phase>process-test-resources</phase>
            <goals>
              <goal>testResources</goal>
            </goals>
            <configuration>
              <encoding>UTF-8</encoding>
            </configuration>
          </execution>
          <execution>
            <id>default-resources</id>
            <phase>process-resources</phase>
            <goals>
              <goal>resources</goal>
            </goals>
            <configuration>
              <encoding>UTF-8</encoding>
            </configuration>
          </execution>
        </executions>
        <dependencies>
          <dependency>
            <groupId>org.apache.maven.shared</groupId>
            <artifactId>maven-filtering</artifactId>
            <version>3.1.1</version>
            <scope>compile</scope>
          </dependency>
        </dependencies>
        <configuration>
          <encoding>UTF-8</encoding>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M5</version>
        <executions>
          <execution>
            <id>default-test</id>
            <phase>test</phase>
            <goals>
              <goal>test</goal>
            </goals>
            <configuration>
              <classesDirectory>C:\Projects\webuicheck\target\classes</classesDirectory>
              <useSystemClassLoader>false</useSystemClassLoader>
              <forkCount>4</forkCount>
              <reuseForks>false</reuseForks>
              <argLine>-Dfile.encoding=UTF-8 -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/1.9.7.M3/aspectjweaver-1.9.7.M3.jar"</argLine>
              <properties>
                <property>
                  <name>listener</name>
                  <value>io.qameta.allure.junit4.AllureJunit4</value>
                </property>
              </properties>
              <systemProperties>
                <property>allure.results.directory</property>
                <value>target/allure-results</value>
              </systemProperties>
            </configuration>
          </execution>
        </executions>
        <dependencies>
          <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>1.9.7.M3</version>
            <scope>compile</scope>
          </dependency>
        </dependencies>
        <configuration>
          <classesDirectory>C:\Projects\webuicheck\target\classes</classesDirectory>
          <useSystemClassLoader>false</useSystemClassLoader>
          <forkCount>4</forkCount>
          <reuseForks>false</reuseForks>
          <argLine>-Dfile.encoding=UTF-8 -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/1.9.7.M3/aspectjweaver-1.9.7.M3.jar"</argLine>
          <properties>
            <property>
              <name>listener</name>
              <value>io.qameta.allure.junit4.AllureJunit4</value>
            </property>
          </properties>
          <systemProperties>
            <property>allure.results.directory</property>
            <value>target/allure-results</value>
          </systemProperties>
        </configuration>
      </plugin>
      <plugin>
        <groupId>io.qameta.allure</groupId>
        <artifactId>allure-maven</artifactId>
        <version>2.10.0</version>
        <configuration>
          <reportVersion>2.13.9</reportVersion>
        </configuration>
      </plugin>
      <plugin>
        <artifactId>maven-clean-plugin</artifactId>
        <version>2.5</version>
        <executions>
          <execution>
            <id>default-clean</id>
            <phase>clean</phase>
            <goals>
              <goal>clean</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-jar-plugin</artifactId>
        <version>2.4</version>
        <executions>
          <execution>
            <id>default-jar</id>
            <phase>package</phase>
            <goals>
              <goal>jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-install-plugin</artifactId>
        <version>2.4</version>
        <executions>
          <execution>
            <id>default-install</id>
            <phase>install</phase>
            <goals>
              <goal>install</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-deploy-plugin</artifactId>
        <version>2.7</version>
        <executions>
          <execution>
            <id>default-deploy</id>
            <phase>deploy</phase>
            <goals>
              <goal>deploy</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-site-plugin</artifactId>
        <version>3.3</version>
        <executions>
          <execution>
            <id>default-site</id>
            <phase>site</phase>
            <goals>
              <goal>site</goal>
            </goals>
            <configuration>
              <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
              <reportPlugins>
                <reportPlugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-project-info-reports-plugin</artifactId>
                </reportPlugin>
              </reportPlugins>
            </configuration>
          </execution>
          <execution>
            <id>default-deploy</id>
            <phase>site-deploy</phase>
            <goals>
              <goal>deploy</goal>
            </goals>
            <configuration>
              <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
              <reportPlugins>
                <reportPlugin>
                  <groupId>org.apache.maven.plugins</groupId>
                  <artifactId>maven-project-info-reports-plugin</artifactId>
                </reportPlugin>
              </reportPlugins>
            </configuration>
          </execution>
        </executions>
        <configuration>
          <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
          <reportPlugins>
            <reportPlugin>
              <groupId>org.apache.maven.plugins</groupId>
              <artifactId>maven-project-info-reports-plugin</artifactId>
            </reportPlugin>
          </reportPlugins>
        </configuration>
      </plugin>
    </plugins>
  </build>
  <reporting>
    <outputDirectory>C:\Projects\webuicheck\target\site</outputDirectory>
  </reporting>
  <profiles>
    <profile>
      <id>firefox</id>
      <properties>
        <browser>firefox</browser>
      </properties>
    </profile>
    <profile>
      <id>chrome</id>
      <properties>
        <browser>chrome</browser>
      </properties>
    </profile>
    <profile>
      <id>ie</id>
      <properties>
        <browser>ie</browser>
      </properties>
    </profile>
  </profiles>
</project>

Answer 4

@stru4OK 正如我之前所说，失败可能是由于目录 target/surefire 中的两个阶段都被覆盖所致。我想我们可以通过设置“tempDir”来解决这个问题，参见https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#tempDir

如果它对您有用，我们可以证实我的假设。所以我们必须试一试。

stage('suites') {
        parallel {
            stage('1 test') {
              sh 'mvn clean test -DtempDir=stage1 -Dtest=FirstSuite'
            }
            stage('2 test') {
              sh 'mvn clean test -DtempDir=stage2 -Dtest=SecondSuite'
            }
        }
}

Answer 5

@tibor17，感谢您的建议！我会检查一下。现在我也在为每个线程测试另一种使用不同目标目录的方法。我在pom.xml中定义了不同的目标目录路径：

 <profiles>
    <profile>
        <id>firstOutputDir</id>
        <build>
            <directory>${project.basedir}/target1/test-classes</directory>
        </build>
    </profile>

    <profile>
        <id>secondOutputDir</id>
        <build>
            <directory>${project.basedir}/target2/test-classes</directory>
        </build>
    </profile>
</profiles>

并在已执行的套件中使用不同的配置文件：

        stage('Tests'){
        parallel {
            stage('FirstSuite'){
                steps {
                    sh 'mvn -e -Dtest=FirstSuite test -PfirstOutputDir'
                }
            }
            stage('SecondSuite'){
                steps {
                    sh 'mvn -e -Dtest=SecondSuite test -PsecondOutputDir'
                }
            }
        }
    }

您如何看待这种做法？在过去的 15-20 次执行中，我没有遇到任何万无一失的错误。也许是巧合，也许是解决方案。不确定）

并行执行测试的周期性问题

Periodic problem with parallel execution of tests

java

jenkins

parallel-processing

maven-surefire-plugin