阶段完成后 Gitlab runner 缓存未命中文件

Question

总结

我的 gitlab-ci.yml 有 3 个阶段用于将应用程序部署到 okd pod 应用程序运行宁 spring 在 tomcat:8 上启动有时，阶段完成后 cache.zip 不会更新，因此下一步无法运行正确

重现步骤

我的gitlab-ci运行下面阶段

阶段 1：运行测试编译 ---> OK

第 2 阶段：打包 war 文件作为部署的输出 ---> Gitlab-ci 日志显示成功但 cache.zip 没有 war 文件（只是有时 cache.zip 没有 war 文件，有时运行正确）

阶段 3：将 war 文件部署到 pod ---> 由于 war 文件在 cache.zip 中不存在，脚本错误 -> 失败

.gitlab-ci.yml

image: openshift/origin-cli

stages:
  - build
  - test
  - staging

cache:
  paths:
    - .m2/repository
    - target
    - artifact

validate:jdk8:
  stage: build
  script:
    - 'mvn test-compile'
  only:
    - master
  image: maven:3.3.9-jdk-8

verify:jdk8:
  stage: test
  script:
    - 'mvn verify'
    - 'mvn package' # =====> this command generate war file
  only:
    - master
  image: maven:3.3.9-jdk-8

staging:
  script:
    - "mkdir -p artifact"
    - "cp ./target/*.war ./artifact/" # ======> Sometimes error at this line because of previous step not add war file into cache
    - "oc start-build $APP"
    - "rm -rf ./target/* && rm -rf ./artifact/*" # Remove war & class file, only cache m2 lib
  stage: staging
  variables:
    APP: $CI_PROJECT_NAME
  environment:
    name: staging
    url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
  only:
    - master

实际行为 有时缓存在测试阶段完成后没有 war 文件（这取决于 war 文件大小吗？）

预期行为 War 暂存阶段部署的测试阶段后将文件更新到缓存中

相关日志and/or截图

ScreenShot

作业日志

Running with gitlab-runner 13.7.0 (943fc252)
  on gitlab-runner-node1 y6awygsj
Preparing the "docker" executor
00:01
Using Docker executor with image openshift/origin-cli ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:7ebb6be01117a50344d63f77c385a13302afecd33480b97c36a518d4f5ebc25a for openshift/origin-cli with digest docker.io/openshift/origin-cli@sha256:509e052d0f2d531b666b7da9fa49c5558c76ce5d286456f0859c0a49b16d6bf2 ...
Preparing environment
00:00
Running on runner-y6awygsj-project-489-concurrent-0 via gitlab.runner.node1...
Getting source from Git repository
00:01
Fetching changes...
Reinitialized existing Git repository in /builds/my-project/.git/
Checking out b4c97428 as master...
Removing .m2/
Removing artifact/
Removing target/
Skipping Git submodules setup
Restoring cache
00:05
Checking cache for default-23...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:01
$ mkdir -p artifact
$ cp ./target/*.war ./artifact/
cp: cannot stat './target/*.war': No such file or directory
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

环境描述

config.toml

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-runner-node1"
  url = "https://gitlab.mycompany.vn/"
  token = "y6awygsj9zks18nU6PDt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    dns = ["192.168.100.1"]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/mnt/nfs/nfsshare-gitlab/cache:/cache"]
    shm_size = 0
    pull_policy = "if-not-present"

已使用 GitLab Runner 版本

版本：13.7.0

Git修订版：943fc252

Git 分支：13-7-稳定

GO版本：go1.13.8

建造时间：2020-12-21T13:47:06+0000

OS/Arch: linux/amd64

可能的修复

重新运行测试阶段直到缓存有war个文件

Answer 1

让我们一步一步来。

首先，关于如何管理阶段之间的文件。

确实，如果运行在同一环境中，您可以直接访问作业和阶段之间的文件，但情况并非总是如此（即使两个运行用户都在使用相同的 nfs 共享目录），您应该为此使用 artifacts。

当您在作业中定义 artifact 时，您指定的是在作业成功、失败或总是时附加到作业的文件列表，具体取决于您的配置。

默认情况下，前面阶段的所有 artifacts 都会传递给每个作业，但在任何情况下，您都可以使用 dependencies 来定义要从中获取工件的作业。

所以基本上你应该使用下面的.gitlab-ci.yml

image: openshift/origin-cli

stages:
  - build
  - test
  - staging

cache:
  paths:
    - .m2/repository

validate:jdk8:
  stage: build
  script:
    - 'mvn test-compile'
  only:
    - master
  image: maven:3.3.9-jdk-8

verify:jdk8:
  stage: test
  script:
    - 'mvn verify' # =====> verify already includes: validate, compile, test and package
  artifacts:
    paths:
      - target/[YOUR_APP_NAME].war
  only:
    - master
  image: maven:3.3.9-jdk-8

staging:
  dependencies:
    - verify:jdk8
  script:
    - "mkdir -p artifact"
    - "cp ./target/[YOUR_APP_NAME].war ./artifact/"
    - "oc start-build $APP"
  stage: staging
  variables:
    APP: $CI_PROJECT_NAME
  environment:
    name: staging
    url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
  only:
    - master

另外，请注意我删除了 mvn package 指令。我建议您查看 Build Lifecycle Basics of Maven.

阶段完成后 Gitlab runner 缓存未命中文件

Gitlab runner cache miss file after stage complete

spring

gitlab

gitlab-ci

gitlab-runner