Cloud Build Bazel Error: "kubectl toolchain was not properly configured so apply cannot be executed"

Question

我正在尝试使用 rules_k8s 将 Bazel 部署到我的 Kubernetes 集群。

因此我有这个 cloudbuild.yaml 文件，它由 Google Cloud Build 执行：

steps:
  - name: gcr.io/cloud-builders/bazel
    args: ['run', '//:kubernetes.apply']

(//:kubernetes 只是一个 k8s_objects)

在我的本地机器上运行 bazel run //:kubernetes.apply 工作正常，但是尽管 Google Cloud Build 成功，但它会记录这些错误。所以配置没有应用到我的 Kubernetes 集群：

Target //:kubernetes.apply up-to-date:
  bazel-bin/kubernetes.apply
INFO: Elapsed time: 29.863s, Critical Path: 0.14s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/kubernetes.apply
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 1 total action
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so projection_database_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so k8s_service.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_deployment.apply cannot be executed.
kubectl toolchain was not properly configured so event_store_k8s_service.apply cannot be executed.

我还从 bazel 缓存中收到警告：

DEBUG: /builder/home/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/io_bazel_rules_k8s/toolchains/kubectl/kubectl_toolchain.bzl:28:9: No kubectl tool was found or built, executing run for rules_k8s targets might not work.

P.S.: 我在使用 //:kubernetes.create

时遇到同样的错误

我的设置

部署

load("@io_bazel_rules_k8s//k8s:object.bzl", "k8s_object")
k8s_object(
  name = "k8s_deployment",
  kind = "deployment",
  cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
  template = ":ideas.deployment.yaml",
  images = {
    "gcr.io/cents-ideas/ideas:latest": ":image"
  },
)

服务

k8s_object(
  name = "k8s_service",
  kind = "service",
  cluster = "gke_cents-ideas_europe-west3-a_cents-ideas",
  template = ":ideas.service.yaml",
)

聚合

load("@io_bazel_rules_k8s//k8s:objects.bzl", "k8s_objects")
k8s_objects(
    name = "k8s",
    objects = [
      ":k8s_deployment",
      ":k8s_service",
    ]
)

最终构图

k8s_objects(
    name = "kubernetes",
    objects = [
        "//services/ideas:k8s",
        # ...
    ]
)

更新

我现在尝试使用 Bazel 和 kubectl 制作我自己的 docker 图像：

FROM gcr.io/cloud-builders/bazel

RUN curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
RUN chmod +x ./kubectl
RUN mv ./kubectl /usr/local/bin/kubectl

我将它推送到 GCR 并将我的 cloudbuild.yaml 更改为：

steps:
  - name: eu.gcr.io/cents-ideas/bazel-kubectl
    args: ["run", "//:kubernetes.apply"]

我首先注意到，这一步比以前花费了更长的时间。但是最后它抛出一个错误：

$ /usr/local/bin/kubectl --kubeconfig= --cluster=gke_cents-ideas_europe-west3-a_cents-ideas --context= --user= apply -f -
error: cluster "gke_cents-ideas_europe-west3-a_cents-ideas" does not exist

Here 是完整日志。

Answer 1

它抱怨在您的机器上找不到 kubectl。如果使用GKE，还需要安装gcloud sdk

还要检查是否为这些工具配置了身份验证：kubectl auth, GKE auth。

Answer 2

主要问题是 kubectl 没有随 gcr.io/cloud-builders/bazel docker 图片一起提供。这就是找不到它的原因。

然而，有人努力将 kubectl 工具链手动实施到 Bazel 中：https://github.com/bazelbuild/rules_k8s/tree/master/toolchains/kubectl#kubectl-toolchain。但由于这些功能目前处于实验状态，它们通常不起作用。

更多信息可以在这个问题中找到：https://github.com/bazelbuild/rules_k8s/issues/512

For now, your best bet will be to install kubectl in the container.

Answer 3

至于更新后的问题，现在您需要以某种方式向容器内的 GKE 进行身份验证。

首先，我建议将 gcloud 工具安装到您的容器中。顺便说一句，至于 1.2 GB 的巨大容器，那是因为 cloud-builders/bazel 很大 :)

看看我们在 slim bazel 容器版本上的例子： https://github.com/aspect-development/bazel-k8s-example/blob/master/tools/Dockerfile.dazel

这是用于安装 gcloud 和 kubectl 的 Dockerfile，因此您可以从这两个文件中获取所需的部分： https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/gcloud/Dockerfile

第二件事是认证，安装gcloud后应该很容易。整个 cloudbuild 步骤应该与此类似：

- name: <link to your container>
  entrypoint: /bin/sh
  args:
  - -c
  - |
    gcloud container clusters get-credentials cents-ideas --zone europe-west3-a --project cents-ideas
    bazel run //:kubernetes.apply

Cloud Build Bazel Error: "kubectl toolchain was not properly configured so apply cannot be executed"

Cloud Build Bazel Error: "kubectl toolchain was not properly configured so apply cannot be executed"

kubernetes

bazel

kubectl

google-cloud-build

我的设置

更新