发布安装由 setup.py 创建的用于将自定义预测部署到 AI 平台的 zip 文件

Question

正在关注 google 关于创建自定义预测的文档 (https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines)。在为模型 AI 平台预测构建新版本时 API 抛出以下错误：

Error Create Version failed. Bad model detected with error: "Failed to load model: User-provided package <>-0.1.tar.gz failed to install: Command '['python-default', '-m', 'pip', 'install', '--target=/tmp/custom_lib', '--no-cache-dir', '-b', '/tmp/pip_builds', '/tmp/custom_code/<>.tar.gz']' returned non-zero exit status 1. (Error code: 0)"

正在本地测试我的 zip 文件

pip install --target=/tmp/custom_lib --no-cache-dir -b /tmp/pip_builds dist/<>

抛出以下错误：

Processing ./dist/<tar>
    ERROR: Command errored out with exit status 1:
     command: bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-req-build-9dmmje_z/setup.py'"'"'; __file__='"'"'/private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-req-build-9dmmje_z/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-pip-egg-info-ic3qhcpo
         cwd: /private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-req-build-9dmmje_z/
    Complete output (6 lines):
    Parent directory: /private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-req-build-9dmmje_z
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-req-build-9dmmje_z/setup.py", line 22, in <module>
        REQUIRED_PACKAGES = [line.strip() for line in open(base/"requirements.txt")] + \
    FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/mj/b_g4b4yj26q278g9k58w92fck5d9wj/T/pip-req-build-9dmmje_z/requirements.txt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

运行 mac 终端上的“python setup.py egg_info”，没有弹出任何有用信息。

Parent directory: .

Packages: ['numpy==1.17.3', 'pandas==1.2.1', 'setuptools==53.0.0', 'Pillow==6.2.1', 'torchvision==0.5.0', 'torch @ https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl']


running egg_info
writing <>.egg-info/PKG-INFO
writing dependency_links to <>.egg-info/dependency_links.txt
writing requirements to <>.egg-info/requires.txt
writing top-level names to <>.egg-info/top_level.txt
reading manifest file '<>.egg-info/SOURCES.txt'
writing manifest file '<>.egg-info/SOURCES.txt'

Python 个包裹：

Packages: ['numpy==1.17.3', 'pandas==1.2.1', 'setuptools==53.0.0', 'Pillow==6.2.1', 'torchvision==0.5.0', 'torch @ https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl']

Setup.py

from pathlib import Path
"""
Purpose: Package your Predictor and its dependencies by packaging  Predictor as a .tar.gz.

For deploying any PyTorch model, the PyTorch package is needed to be provided in the below-mentioned command. In this case, 
we have used the package:
torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl
For getting pytorch packages according to your requirements,
Visit this link: Pytorch Packages(google search)
Download the package you require and store it in the GCS bucket.
The package we are using provides version resource with PyTorch 1.3.1 for Python 3.7, built to run on a CPU in Linux.

NOTE: We could have directly added ‘torch’ in the ‘REQUIRED_PACKAGES’ list in setup.py file in order to provide pytorch as a dependency to be installed while deployment.
This way, it uses the PyPI installation of pytorch.
But the issue with this approach is that it internally downloads PyPI package for pytorch which is of 720 MB Which causes the failure of our model deployment because the AI platform allows custom models of 500MB or below to be deployed on it.
So it is recommended to provide a pytorch package manually using gsutil command under — package-uris.
Try: pip install --target=/tmp/custom_lib --no-cache-dir -b /tmp/pip_builds <>.4.tar.gz for troubleshooting
"""
base = Path(__file__).parent
print(f"Parent directory: {base}")
REQUIRED_PACKAGES = [line.strip() for line in open(base/"requirements.txt")] + \
['torchvision==0.5.0', 'torch @ https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl']

print(f"\nPackages: {REQUIRED_PACKAGES}\n\n")

setup(description="Extract features of a image",
      author=<>,
      author_email=<>,
      name=<>,
      version=<>,
      url='<>',
      install_requires=REQUIRED_PACKAGES,
      project_urls={
          'Documentation': 'https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines#tensorflow',
          'Deploy': 'https://cloud.google.com/ai-platform/prediction/docs/deploying-models#gcloud_1',
          'Ai_platform troubleshooting': 'https://cloud.google.com/ai-platform/training/docs/troubleshooting',
          'Say Thanks!': ['https://medium.com/searce/deploy-your-own-custom-model-on-gcps-ai-platform-7e42a5721b43',
                          '
                          '
          'google Torch wheels': "http://storage.googleapis.com/cloud-ai-pytorch/readme.txt",
          'Torch & torchvision wheels': "https://download.pytorch.org/whl/torch_stable.html ",
          'Raised issue with google github branch': "https://github.com/GoogleCloudPlatform/python-docs-samples/issues/5257",
          "Whosebug": "
      },
    python_requires='~=3.7',
    scripts=[<>])```
Have tried the usual upgrading python setuptools to 52.0.0, but can't seem to shake this error off.

Answer 1

我能够通过删除 'requirement.txt' 并将包直接复制到 setup.py 来使其工作。不知道这种奇怪行为的真正原因。

发布安装由 setup.py 创建的用于将自定义预测部署到 AI 平台的 zip 文件

Issue installing zip file created by setup.py for deploying custom prediction to AI platform

setup.py

google-cloud-ml

google-ai-platform