无法从 docker 构建中的工件注册表安装私有依赖项
Cannot install private dependency from artifact registry inside docker build
我正在尝试安装一个私有 python 包,该包已上传到 docker 容器内的工件注册表(以将其部署在 cloudrun 上)。
我过去曾在云函数中成功使用过该包,因此我确信该包有效。
cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']
Dockerfile
FROM python:3.8.6-slim-buster
ENV APP_PATH=/usr/src/app
ENV PORT=8080
# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython
RUN pip install --upgrade pip
# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH
COPY / $APP_PATH
WORKDIR $APP_PATH
RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs
# Expose port
EXPOSE $PORT
# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app
我添加的权限是:
- cloudbuild 默认服务帐户(项目编号@cloudbuild.gserviceaccount.com):Artifact Registry Reader
- 服务帐号运行cloudbuild:Artifact RegistryReader
- 服务帐户运行应用程序:Artifact Registry Reader
cloudbuild错误:
Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
从您的回溯日志中,我们可以看到 Cloud Build 没有向私有存储库进行身份验证的凭据:
Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME
我将一个简单的包上传到私有 Artifact Registry 存储库以在构建容器时对此进行测试,并且也收到了相同的消息。由于您似乎正在使用 service account key 进行身份验证,因此用户名和密码需要存储在 pip.conf
:
中
pip.conf
[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/
因此,此文件需要在构建过程中可用。 Multi-stage docker 构建在这里非常有用,可确保配置密钥不会暴露,因为我们可以选择将哪些文件放入最终映像(配置密钥仅在用于从私人回购):
示例 Dockerfile
# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build
# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf
WORKDIR /packages/
COPY requirements.txt /
# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf
# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES
# Final image that will be deployed
FROM python:3.8.6-slim-buster
ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/
# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*
WORKDIR $APP_HOME
COPY ./src/main.py ./
# Executing sample flask web app
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
我上面的 docker 文件基于这个 ,我可以确认这些包是从我的私人 Artifact Registry 存储库正确下载的,而且 pip.conf
文件是结果图像中不存在。
我正在尝试安装一个私有 python 包,该包已上传到 docker 容器内的工件注册表(以将其部署在 cloudrun 上)。
我过去曾在云函数中成功使用过该包,因此我确信该包有效。
cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'gcr.io/${_PROJECT}/${_SERVICE_NAME}:$SHORT_SHA', '--network=cloudbuild', '.', '--progress=plain']
Dockerfile
FROM python:3.8.6-slim-buster
ENV APP_PATH=/usr/src/app
ENV PORT=8080
# Copy requirements.txt to the docker image and install packages
RUN apt-get update && apt-get install -y cython
RUN pip install --upgrade pip
# Set the WORKDIR to be the folder
RUN mkdir -p $APP_PATH
COPY / $APP_PATH
WORKDIR $APP_PATH
RUN pip install -r requirements.txt --no-color
RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3 # This line is where the bug occurs
# Expose port
EXPOSE $PORT
# Use gunicorn as the entrypoint
CMD exec gunicorn --bind 0.0.0.0:8080 app:app
我添加的权限是:
- cloudbuild 默认服务帐户(项目编号@cloudbuild.gserviceaccount.com):Artifact Registry Reader
- 服务帐号运行cloudbuild:Artifact RegistryReader
- 服务帐户运行应用程序:Artifact Registry Reader
cloudbuild错误:
Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
status = run_func(*args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
return func(self, options, args)
File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 340, in run
requirement_set = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
result = self._result = resolver.resolve(
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
if not criterion.candidates:
File "/usr/local/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
从您的回溯日志中,我们可以看到 Cloud Build 没有向私有存储库进行身份验证的凭据:
Step 10/12 : RUN pip install --extra-index-url https://us-west1-python.pkg.dev/my-project/my-package/simple/ my-package==0.2.3
---> Running in b2ead00ccdf4
Looking in indexes: https://pypi.org/simple, https://us-west1-python.pkg.dev/muse-speech-devops/gcp-utils/simple/
User for us-west1-python.pkg.dev: [91mERROR: Exception: //<-ASKING FOR USERNAME
我将一个简单的包上传到私有 Artifact Registry 存储库以在构建容器时对此进行测试,并且也收到了相同的消息。由于您似乎正在使用 service account key 进行身份验证,因此用户名和密码需要存储在 pip.conf
:
pip.conf
[global]
extra-index-url = https://_json_key_base64:KEY@LOCATION-python.pkg.dev/PROJECT/REPOSITORY/simple/
因此,此文件需要在构建过程中可用。 Multi-stage docker 构建在这里非常有用,可确保配置密钥不会暴露,因为我们可以选择将哪些文件放入最终映像(配置密钥仅在用于从私人回购):
示例 Dockerfile
# Installing packages in a separate image
FROM python:3.8.6-slim-buster as pkg-build
# Target Python environment variable to bind to pip.conf
ENV PIP_CONFIG_FILE /pip.conf
WORKDIR /packages/
COPY requirements.txt /
# Copying the pip.conf key file only during package downloading
COPY ./config/pip.conf /pip.conf
# Packages are downloaded to the /packages/ directory
RUN pip download -r /requirements.txt
RUN pip download --extra-index-url https://LOCATION-python.pkg.dev/PROJECT/REPO/simple/ PACKAGES
# Final image that will be deployed
FROM python:3.8.6-slim-buster
ENV PYTHONUNBUFFERED True
ENV APP_HOME /app
WORKDIR /packages/
# Copying ONLY the packages from the previous build
COPY --from=pkg-build /packages/ /packages/
# Installing the packages from the copied files
RUN pip install --no-index --find-links=/packages/ /packages/*
WORKDIR $APP_HOME
COPY ./src/main.py ./
# Executing sample flask web app
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app
我上面的 docker 文件基于这个 pip.conf
文件是结果图像中不存在。