如何使用 tensorflow-nightly 和 tensorflow 对象检测研究模型构建 docker 图像

Question

因为 GPU 对 tensorflow-nightly 的支持是 currently broken on Google Colab I'm trying to build my own docker image for development. However, when I install the object_detection package from tensorflow/models my nightly tensorflow package is overwritten by the version pulled in as a dependency from the object_detection setup.py。

我在 Google Colab 中遵循基本相同的步骤，但我的 tensorflow nightly 没有在那里被覆盖，所以我不确定我错过了什么...

这是我的 Dockerfile:

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

我正在构建的：

docker pull tensorflow/tensorflow:nightly-gpu-jupyter
docker build --no-cache . -f models-tf-nightly.Dockerfile -t tf-nightly-models

第一个print()显示：

Tensorflow version: 2.5.0-dev20201129

但是第二个显示：

Tensorflow version: 2.3.1

在 Google Colab 中，我基本上执行相同的步骤：

# Install the Object Detection API
%%bash
pip install tf-nightly-gpu
[[ -d models ]] || git clone --depth 1 https://github.com/tensorflow/models
cd models/research/
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .

之后

import tensorflow as tf
print(tf.__version__)

打印2.5.0-dev20201201

所以我的 Google Colab 步骤以某种方式保留了我每晚安装的 Tensorflow，而在 Docker 上它被 2.3.0 覆盖。

Answer 1

如果您在安装对象检测包之前查看 pip list，您会看到 tf-nightly-gpu 已安装，但 tensorflow 未安装。当您安装对象检测包时，tensorflow 包作为依赖项被引入。 pip认为没有安装，所以安装了。

解决此问题的一种方法是欺骗 pip install 认为 tensorflow 软件包已安装。可以通过符号链接 dist-packages 中的 tf_nightly_gpu-VERSION.dist-info 目录来做到这一点。我在下面的 Dockerfile 中添加了执行此操作的行。在此 post 的底部，我还包含了一个 Dockerfile，它实施了一些最佳实践以最小化图像大小。

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)//')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

这里是一个 Dockerfile，它导致一个稍微小一点的图像（0.22 GB 未压缩）。显着的变化是清除 apt 列表并在 pip install.

中使用 --no-cache-dir

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        git \
        less \
        zip && \
    rm -rf /var/lib/apt/lists/*

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)//')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && \
    unzip protoc-3.11.4-linux-x86_64.zip && \
    cp bin/protoc /usr/local/bin && \
    rm -r protoc-3.11.4-linux-x86_64.zip bin/

# Upgrade pip.
RUN python -m pip install --no-cache-dir --upgrade pip

RUN git clone --depth 1 https://github.com/tensorflow/models
WORKDIR models/research
RUN protoc object_detection/protos/*.proto --python_out=. && \
    cp object_detection/packages/tf2/setup.py . && \
    python -m pip install  --no-cache-dir .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

如何使用 tensorflow-nightly 和 tensorflow 对象检测研究模型构建 docker 图像

How to build a docker image with tensorflow-nightly and the tensorflow object detection research models

python

pip

docker

tensorflow

tensorflow-model-garden