从云函数内部构建容器镜像

Question

上下文：我在 Google Vertex AI 中为每个 bigquery 数据集训练一个非常相似的模型，但我想为每个现有数据集（在 Google BigQuery 中）都有一个自定义训练图像。从这个意义上讲，我需要按需在容器注册表中以编程方式构建自定义 Docker 图像。我的想法是让 Google Cloud Function 执行此操作，由 PubSub 主题触发，其中包含有关我要为其构建训练容器的数据集的信息。因此，该函数自然会将 Docker 文件和相关脚本写入 Cloud Functions 中的 /tmp 文件夹（据我所知，这是唯一可写的地方）。但是，当我尝试在此脚本中实际构建容器时，显然它找不到 /tmp 文件夹或其内容，即使它们在那里（通过日志记录操作检查）。

到目前为止麻烦的代码：

def build_container(dataset=str):

    with open('container_template/Dockerfile','r') as f:
        dockerfile = f.read()
    dockerfile = dockerfile.replace('@dataset',dataset)
    f.close()

    os.makedirs(os.path.dirname('/tmp/script-location'), exist_ok=True)

    with open('/tmp/Dockerfile','w') as docker_config:
        docker_config.write(dockerfile)

    docker_config.close()

    shutil.copy('container_template/script-location/script.py','/tmp/script-location/script.py')

    build_client = cloudbuild_v1.CloudBuildClient()

    build = cloudbuild_v1.Build()

    build.steps = [{'name':'gcr.io/cloud-builders/docker',
                    'args':['build', '-t', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest','/tmp']},
                    {'name':'gcr.io/cloud-builders/docker',
                    'args':['push', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest']}]  

    build_operation = build_client.create_build(project_id=myprojectid,build=build)

    build_result = build_operation.result()

    logger.info('Build Result: {}'.format(build_result.status))

当我检查云构建日志时，我得到：步骤#0：无法准备上下文：无法评估 Docker 文件路径中的符号链接：lstat /tmp/Dockerfile：没有这样的文件或目录

Answer 1

我已经在本地测试了使用 Cloud Build Client Python library 构建容器映像。事实证明即使 Dockerfile 文件存在于当前目录中也会出现相同的错误：

错误：

Step #0: unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /workspace/Dockerfile: no such file or directory

构建步骤：

    build_client = cloudbuild_v1.CloudBuildClient()

    build = cloudbuild_v1.Build()

    build.steps = [{'name':'gcr.io/cloud-builders/docker',
                    'args':['build', '-t', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest','.']},
                    {'name':'gcr.io/cloud-builders/docker',
                    'args':['push', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest']}]  

    build_operation = build_client.create_build(project_id=myprojectid,build=build)

    build_result = build_operation.result()

因为它使用了 API 方法，所以我遵循了这个 documentation. You will see the source is present in API method. It is the missing key to move forward the problem. In StorageSource，你必须指定 bucket 和 object_。您需要压缩源代码并将其上传到 Cloud Storage 存储桶中。例如：

运行以下命令压缩你的源代码:

tar -cvzf sourcecode.tar.gz .

上传到 Cloud Storage 存储桶（您可以使用 Cloud Build 存储桶）：

gsutil cp sourcecode.tar.gz gs://myproject_cloudbuild

build.source:

    build_client = cloudbuild_v1.CloudBuildClient()

    build = cloudbuild_v1.Build()
    
    build.source = {"storage_source":{"bucket":"myproject_cloudbuild", "object_":"gs://myproject_cloudbuild/sourcecode.tar.gz"}}

    build.steps = [{'name':'gcr.io/cloud-builders/docker',
                    'args':['build', '-t', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest','.']},
                    {'name':'gcr.io/cloud-builders/docker',
                    'args':['push', 'us-central1-docker.pkg.dev/myproject/myrepo/imagename:latest']}]  

    build_operation = build_client.create_build(project_id=myprojectid,build=build)

    build_result = build_operation.result()

因此它解决了使用客户端库构建图像的问题。我建议在您的 Cloud Function 中执行所有这些操作。

从云函数内部构建容器镜像

Build a container image from inside a cloud function

python

docker

google-cloud-functions

google-cloud-build

google-artifact-registry