Gitlab CI 使用 dockerd gitlab-runner 在 gradle 多模块上可视化问题

Question

卓悦，

大家好，希望一切顺利。

过去几天，我使用 gitlab-CI 和 jacoco 测试报告以及 cobertura 转换分析在 multi-module gradle 项目上使用 gitlab 实例设置覆盖率可视化.并且在为可视化任务设置 gitlab-CI 时需要一点帮助。

我当前的gitlab-ci.yaml文件如下：

# This file is a template, and might need editing before it works on your project.
# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Gradle.gitlab-ci.yml

# This is the Gradle build system for JVM applications
# https://gradle.org/
# https://github.com/gradle/gradle

image: gradle:latest

# Disable the Gradle daemon for Continuous Integration servers as correctness
# is usually a priority over speed in CI environments. Using a fresh
# runtime for each build is more reliable since the runtime is completely
# isolated from any previous builds.
variables:
  GRADLE_OPTS: "-Dorg.gradle.daemon=false"

stages:
  - build
  - test
  - visualize

before_script:
  - export GRADLE_USER_HOME=`pwd`/.gradle

cache:
  paths:
    - .gradle/wrapper
    - .gradle/caches

build:
  stage: build
  script: gradle -g /cache --build-cache assemble
  cache:
    key: "$CI_COMMIT_REF_NAME"
    policy: push
    paths:
      - build
      - .gradle

test:
  stage: test
  script: gradle -g /cache test jacocoTestReport
  cache:
    key: "$CI_COMMIT_REF_NAME"
    policy: pull
    paths:
      - build
      - .gradle
  after_script:
    - mkdir build
    - cat reports/tests/test/index.html | grep -o '<tfoot>.*</tfoot>'
  artifacts:
    when: always
    reports:
      junit: "**/build/test-results/test/**/TEST-*.xml"
    # paths:
    #   - "jacoco-aggregate/jacoco.xml"

coverage:
  stage: visualize
  image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7
  cache:
    key: "$CI_COMMIT_REF_NAME"
    policy: pull
    paths:
      - build
      - .gradle
  script:
    # find all modules containing java source files.
    - jacoco_paths=`find * -path "**/src/main/java" -type d | sed -e 's@^@'"$CI_PROJECT_DIR"'/@'`
    # convert report from jacoco to cobertura, using relative project path
    - python /opt/cover2cover.py reports/jacoco.xml $jacoco_paths > $CI_PROJECT_DIR/build/coverage.xml
  needs: ["test", "build"]
  dependencies:
    - test
  artifacts:
    reports:
      cobertura: "$CI_PROJECT_DIR/build/coverage.xml"

我遇到的问题是，当 CI stage 运行时，它会清除所有以前的构建目录，也无法从缓存中提取，并且管道失败。而且已经失败太多次了。

我的 Gitlab-runner 在 docker 容器上。

CI 控制台日志：

Running with gitlab-runner 14.5.2 (e91107dd)
  on docker E9Kbrzec
Preparing the "docker" executor
00:06
Using Docker executor with image registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 ...
Pulling docker image registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 ...
Using docker image sha256:8d5913f8379757d81e4ec8a21308162acb805e0e875d1d157702983baa38f5ba for registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 with digest registry.gitlab.com/haynes/jacoco2cobertura@sha256:1b8ee364eba07c074f907c9e28ade778671e437e09b5e4457849979fb4265065 ...
Preparing environment
00:02
Running on runner-e9kbrzec-project-2-concurrent-0 via 0b8d013e9fb0...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/chirag/example/.git/
Checking out 954c86a7 as ci-coverage...
Removing .gradle/
Removing build/
Removing example-api/build/
Removing example-domain/build/
Removing example-services/build/
Removing example-services/data/
Removing example-services/temp/
Removing example-web/build/
Skipping Git submodules setup
Restoring cache
00:02
Checking cache for ci-coverage...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:02
Using docker image sha256:8d5913f8379757d81e4ec8a21308162acb805e0e875d1d157702983baa38f5ba for registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 with digest registry.gitlab.com/haynes/jacoco2cobertura@sha256:1b8ee364eba07c074f907c9e28ade778671e437e09b5e4457849979fb4265065 ...
$ export GRADLE_USER_HOME=`pwd`/.gradle
$ jacoco_paths=`find * -path "**/src/main/java" -type d | sed -e 's@^@'"$CI_PROJECT_DIR"'/@'`
$ python /opt/cover2cover.py reports/jacoco.xml $jacoco_paths > $CI_PROJECT_DIR/build/coverage.xml
Traceback (most recent call last):
  File "/opt/cover2cover.py", line 163, in <module>
    jacoco2cobertura(filename, source_roots)
  File "/opt/cover2cover.py", line 147, in jacoco2cobertura
    tree = ET.parse(filename)
  File "/usr/local/lib/python3.8/xml/etree/ElementTree.py", line 1202, in parse
    tree.parse(source, parser)
  File "/usr/local/lib/python3.8/xml/etree/ElementTree.py", line 584, in parse
    source = open(source, "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'reports/jacoco.xml'
Uploading artifacts for failed job
00:02
Uploading artifacts...
/builds/chirag/example/build/coverage.xml: found 1 matching files and directories 
Uploading artifacts as "cobertura" to coordinator... ok  id=138 responseStatus=201 Created token=jCWt186i
ERROR: Job failed: exit code 1

日志也显示缓存已恢复，但在脚本上写入 ls 仅显示 src 文件夹。

请协助我设置覆盖作业。

提前致谢。

Answer 1

我最终开发了自己的插件 jacotura-gradle-plugin，将 jacoco 报告转换为 cobertura 报告并作为 gradle 任务执行。

参考：Jacotura @ Github

Gitlab CI 使用 dockerd gitlab-runner 在 gradle 多模块上可视化问题

Gitlab CI visualize issues on gradle multi-module with dockerd gitlab-runner

gitlab-ci

jacoco

cobertura

cicd

gitlab-pipelines