Google Cloud Build - Terraform 在构建失败时自毁

Question

我的 Google Cloud Build for CI/CD 目前遇到问题。

首先，我构建了多个微服务的新 docker 映像，并使用 Terraform 为容器创建 GCP 基础设施，它们也将在生产环境中使用。
然后我执行一些集成/系统测试，如果一切正常，我将新版本的微服务映像推送到容器注册表以供以后部署。

我的问题是，如果云构建失败，Terraformed 基础设施不会被破坏。

有没有办法始终执行云构建步骤，即使之前的某些步骤失败了，在这里我想始终执行“terraform destroy”？或者专门针对 Terraform，有没有办法定义一个自毁的 Terraform 环境？

cloudbuild.yaml 仅包含一个 docker 容器的示例

steps:

    # build fresh ...
    - id: build 
      name: 'gcr.io/cloud-builders/docker'
      dir: '...'
      args: ['build', '-t', 'gcr.io/$PROJECT_ID/staging/...:latest', '-t', 'gcr.io/$PROJECT_ID/staging/...:$BUILD_ID', '.', '--file', 'production.dockerfile']

    # push
    - id: push
      name: 'gcr.io/cloud-builders/docker'
      dir: '...'
      args: ['push', 'gcr.io/$PROJECT_ID/staging/...']
      waitFor: [build]

    # setup terraform
    - id: terraform-init
      name: 'hashicorp/terraform:0.12.28'
      dir: '...'
      args: ['init']
      waitFor: [push]

    # deploy GCP resources
    - id: terraform-apply 
      name: 'hashicorp/terraform:0.12.28'
      dir: '...'
      args: ['apply', '-auto-approve']
      waitFor: [terraform-init]

    # tests
    - id: tests
      name: 'python:3.7-slim'
      dir: '...'
      waitFor: [terraform-apply]
      entrypoint: /bin/sh
      args:
       - -c
       - 'pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate'

    # remove GCP resources
    - id: terraform-destroy
      name: 'hashicorp/terraform:0.12.28'
      dir: '...'
      args: ['destroy', '-auto-approve']
      waitFor: [tests]

Answer 1

Google Cloud Build 尚不支持 allow_failure 或 this unsolved but closed issue.

中提到的某些类似机制

如链接问题中所述，您可以做的是链接 shell 条件运算符。

如果你想运行一个失败的命令，那么你可以这样做：

    - id: tests
      name: 'python:3.7-slim'
      dir: '...'
      waitFor: [terraform-apply]
      entrypoint: /bin/sh
      args:
       - -c
       - pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate || echo "This failed!"

这将运行您的测试正常进行，然后在测试失败时回显 This failed! 到日志。如果您想在失败时运行 terraform destroy -auto-approve，则需要将 echo "This failed!" 替换为 terraform destroy -auto-approve。当然，您还需要使用的 Docker 图像中的 Terraform 二进制文件，因此需要使用同时包含 Python 和 Terraform 的自定义图像才能正常工作。

    - id: tests
      name: 'example-customer-python-and-terraform-image:3.7-slim-0.12.28'
      dir: '...'
      waitFor: [terraform-apply]
      entrypoint: /bin/sh
      args:
       - -c
       - pip install -r requirements.txt && pytest ... --tfstate terraform.tfstate || terraform destroy -auto-approve ; false"

上面的作业也在命令的末尾运行s false 这样它将 return 一个非 0 退出代码并将作业标记为失败而不是仅如果 terraform destroy 也失败，则失败。

另一种方法是使用 Test Kitchen 之类的东西，它会自动建立基础设施，运行必要的验证者，然后在最后销毁它，所有这些都在一个 kitchen test命令。

可能还值得一提的是，您的管道完全是串行的，因此您不需要使用 waitFor。 Google Cloud Build documentation:

中提到了这一点

A build step specifies an action that you want Cloud Build to perform. For each build step, Cloud Build executes a docker container as an instance of docker run. Build steps are analogous to commands in a script and provide you with the flexibility of executing arbitrary instructions in your build. If you can package a build tool into a container, Cloud Build can execute it as part of your build. By default, Cloud Build executes all steps of a build serially on the same machine. If you have steps that can run concurrently, use the waitFor option.

and

Use the waitFor field in a build step to specify which steps must run before the build step is run. If no values are provided for waitFor, the build step waits for all prior build steps in the build request to complete successfully before running. For instructions on using waitFor and id, see Configuring build step order.

Google Cloud Build - Terraform 在构建失败时自毁

Google Cloud Build - Terraform Self-Destruction on Build Failure

google-cloud-platform

terraform

terraform-provider-gcp

google-cloud-build

cloudbuild.yaml 仅包含一个 docker 容器的示例