如何使用 virtualenv 安装包但仍然使用系统 tensorflow 安装
How to use a virtualenv to install packages but still use the system tensorflow install
我在本地机器上设置了它，以便在进行 ML 测试时在虚拟环境中安装这些依赖项
tensorflow
keras
h5py
requests
pillow
tensorflow-hub
我有脚本可以完成，创建一个 venv，安装要求，并执行培训。如果我可以在任何地方使用这些脚本就好了，包括在 Google 深度学习平台 VM 上，但是当我尝试在 venv 中安装 tensorflow 时它不再使用 GPU，所以我假设它不再使用tensorflow的系统安装。
我也尝试了 --system-site-packages 标志，它说 tensorflow 已经安装，但随后它停止使用 GPU。
假设我执行以下操作
$ virtualenv --sysem-site-packages venv/
$ source venv/bin/activate
$ pip install -r requirements.txt
Collecting tensorflow (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/1a/c4/8cb95df0bf06089014259b25997c3921a87aa08e2cd981417d91ca92f7e9/tensorflow-1.10.1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: keras in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 2)) (2.2.2)
Requirement already satisfied: h5py in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 3)) (2.7.0)
Requirement already satisfied: requests in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) (2.12.4)
Requirement already satisfied: pillow in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 5)) (4.0.0)
Requirement already satisfied: tensorflow-hub in /home/john/.local/lib/python2.7/site-packages (from -r requirements.txt (line 6)) (0.1.1)
Collecting numpy<=1.14.5,>=1.13.3 (from tensorflow->-r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/6a/a9/c01a2d5f7b045f508c8cefef3b079fe8c413d05498ca0ae877cffa230564/numpy-1.14.5-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.14.1)
Requirement already satisfied: protobuf>=3.6.0 in /home/john/.local/lib/python2.7/site-packages (from tensorflow->-r requirements.txt (line 1)) (3.6.1)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: backports.weakref>=1.0rc1 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.0.post1)
Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (0.3.0)
Requirement already satisfied: wheel in ./venv/lib/python2.7/site-packages (from tensorflow->-r requirements.txt (line 1)) (0.31.1)
Requirement already satisfied: tensorboard<1.11.0,>=1.10.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.10.0)
Requirement already satisfied: six>=1.10.0 in /home/john/.local/lib/python2.7/site-packages (from tensorflow->-r requirements.txt (line 1)) (1.11.0)
Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (0.2.0)
Requirement already satisfied: mock>=2.0.0 in /usr/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (2.0.0)
Requirement already satisfied: enum34>=1.1.6 in /usr/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.1.6)
Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (0.7.1)
Collecting setuptools<=39.1.0 (from tensorflow->-r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/8c/10/79282747f9169f21c053c562a0baa21815a8c7879be97abd930dbcf862e8/setuptools-39.1.0-py2.py3-none-any.whl
Requirement already satisfied: pyyaml in /usr/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (3.12)
Requirement already satisfied: scipy>=0.14 in /usr/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (0.18.1)
Requirement already satisfied: keras-applications==1.0.4 in /usr/local/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (1.0.4)
Requirement already satisfied: keras-preprocessing==1.0.2 in /usr/local/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (1.0.2)
Requirement already satisfied: futures>=2.2.0 in /usr/local/lib/python2.7/dist-packages (from grpcio>=1.8.6->tensorflow->-r requirements.txt (line 1)) (3.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/lib/python2.7/dist-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow->-r requirements.txt (line 1)) (2.6.8)
Requirement already satisfied: werkzeug>=0.11.10 in /usr/lib/python2.7/dist-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow->-r requirements.txt (line 1)) (0.11.15)
tensorflow-serving-api 1.10.0 has requirement protobuf==3.6.0, but you'll have protobuf 3.6.1 which is incompatible.
Installing collected packages: numpy, setuptools, tensorflow
Found existing installation: numpy 1.15.1
Not uninstalling numpy at /home/john/.local/lib/python2.7/site-packages, outside environment /home/john/retrain/venv
Can't uninstall 'numpy'. No files were found to uninstall.
Found existing installation: setuptools 40.2.0
Uninstalling setuptools-40.2.0:
Successfully uninstalled setuptools-40.2.0
Successfully installed numpy-1.14.5 setuptools-39.1.0 tensorflow-1.10.1
对于 tensorflow 的所有依赖项，它显示存在于系统中，因此它不会安装它们，但无论如何它都会安装 tensorflow。这是为什么？
在您的需求文件中，您列出了 tensorflow 包，这是 CPU-only 包。对于 GPU 支持，请安装 tensorflow-gpu。
不幸的是，没有针对 CPU 和 GPU 进行优化的 "fat" tensorflow 二进制文件。但是，可以在两个实例上使用 tensorflow-gpu。
正在 CPU 实例上安装 tensorflow-gpu
事实上，可以在没有 GPU 的实例上使用 tensoflow-gpu 二进制文件。为了使用它，您需要在实例上安装 CUDA 和 CuDNN（即使该实例没有 Nvidia GPU）。 CUDA，里面有一个模拟（存根）Nvidia 驱动程序，它将允许 CUDA 和 CuDNN 在 CPU 之上工作，为了在 linux 上使用它，您需要 运行 以下命令：
sudo ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/libcuda.so.1
sudo ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/libcuda.so
假设/usr/local/cuda是CUDA的安装路径（不同平台可能不一样）。一旦完成，就可以在 CPU-only 实例上实际安装和使用 tensorflow-gpu。
我知道这看起来像是 hack，甚至可能无法在某些平台上运行，但至少在某种程度上可以在 GPU 和非 GPU 实例上使用相同的 requirenemts.txt 甚至相同的二进制文件。
我在本地机器上设置了它，以便在进行 ML 测试时在虚拟环境中安装这些依赖项
tensorflow
keras
h5py
requests
pillow
tensorflow-hub
我有脚本可以完成，创建一个 venv，安装要求，并执行培训。如果我可以在任何地方使用这些脚本就好了，包括在 Google 深度学习平台 VM 上，但是当我尝试在 venv 中安装 tensorflow 时它不再使用 GPU，所以我假设它不再使用tensorflow的系统安装。
我也尝试了 --system-site-packages 标志，它说 tensorflow 已经安装，但随后它停止使用 GPU。
假设我执行以下操作
$ virtualenv --sysem-site-packages venv/
$ source venv/bin/activate
$ pip install -r requirements.txt
Collecting tensorflow (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/1a/c4/8cb95df0bf06089014259b25997c3921a87aa08e2cd981417d91ca92f7e9/tensorflow-1.10.1-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: keras in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 2)) (2.2.2)
Requirement already satisfied: h5py in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 3)) (2.7.0)
Requirement already satisfied: requests in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) (2.12.4)
Requirement already satisfied: pillow in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 5)) (4.0.0)
Requirement already satisfied: tensorflow-hub in /home/john/.local/lib/python2.7/site-packages (from -r requirements.txt (line 6)) (0.1.1)
Collecting numpy<=1.14.5,>=1.13.3 (from tensorflow->-r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/6a/a9/c01a2d5f7b045f508c8cefef3b079fe8c413d05498ca0ae877cffa230564/numpy-1.14.5-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.14.1)
Requirement already satisfied: protobuf>=3.6.0 in /home/john/.local/lib/python2.7/site-packages (from tensorflow->-r requirements.txt (line 1)) (3.6.1)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: backports.weakref>=1.0rc1 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.0.post1)
Requirement already satisfied: absl-py>=0.1.6 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (0.3.0)
Requirement already satisfied: wheel in ./venv/lib/python2.7/site-packages (from tensorflow->-r requirements.txt (line 1)) (0.31.1)
Requirement already satisfied: tensorboard<1.11.0,>=1.10.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.10.0)
Requirement already satisfied: six>=1.10.0 in /home/john/.local/lib/python2.7/site-packages (from tensorflow->-r requirements.txt (line 1)) (1.11.0)
Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (0.2.0)
Requirement already satisfied: mock>=2.0.0 in /usr/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (2.0.0)
Requirement already satisfied: enum34>=1.1.6 in /usr/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (1.1.6)
Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python2.7/dist-packages (from tensorflow->-r requirements.txt (line 1)) (0.7.1)
Collecting setuptools<=39.1.0 (from tensorflow->-r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/8c/10/79282747f9169f21c053c562a0baa21815a8c7879be97abd930dbcf862e8/setuptools-39.1.0-py2.py3-none-any.whl
Requirement already satisfied: pyyaml in /usr/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (3.12)
Requirement already satisfied: scipy>=0.14 in /usr/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (0.18.1)
Requirement already satisfied: keras-applications==1.0.4 in /usr/local/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (1.0.4)
Requirement already satisfied: keras-preprocessing==1.0.2 in /usr/local/lib/python2.7/dist-packages (from keras->-r requirements.txt (line 2)) (1.0.2)
Requirement already satisfied: futures>=2.2.0 in /usr/local/lib/python2.7/dist-packages (from grpcio>=1.8.6->tensorflow->-r requirements.txt (line 1)) (3.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/lib/python2.7/dist-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow->-r requirements.txt (line 1)) (2.6.8)
Requirement already satisfied: werkzeug>=0.11.10 in /usr/lib/python2.7/dist-packages (from tensorboard<1.11.0,>=1.10.0->tensorflow->-r requirements.txt (line 1)) (0.11.15)
tensorflow-serving-api 1.10.0 has requirement protobuf==3.6.0, but you'll have protobuf 3.6.1 which is incompatible.
Installing collected packages: numpy, setuptools, tensorflow
Found existing installation: numpy 1.15.1
Not uninstalling numpy at /home/john/.local/lib/python2.7/site-packages, outside environment /home/john/retrain/venv
Can't uninstall 'numpy'. No files were found to uninstall.
Found existing installation: setuptools 40.2.0
Uninstalling setuptools-40.2.0:
Successfully uninstalled setuptools-40.2.0
Successfully installed numpy-1.14.5 setuptools-39.1.0 tensorflow-1.10.1
对于 tensorflow 的所有依赖项，它显示存在于系统中，因此它不会安装它们，但无论如何它都会安装 tensorflow。这是为什么？
在您的需求文件中，您列出了 tensorflow 包，这是 CPU-only 包。对于 GPU 支持，请安装 tensorflow-gpu。
不幸的是，没有针对 CPU 和 GPU 进行优化的 "fat" tensorflow 二进制文件。但是，可以在两个实例上使用 tensorflow-gpu。
正在 CPU 实例上安装 tensorflow-gpu
事实上，可以在没有 GPU 的实例上使用 tensoflow-gpu 二进制文件。为了使用它，您需要在实例上安装 CUDA 和 CuDNN（即使该实例没有 Nvidia GPU）。 CUDA，里面有一个模拟（存根）Nvidia 驱动程序，它将允许 CUDA 和 CuDNN 在 CPU 之上工作，为了在 linux 上使用它，您需要 运行 以下命令：
sudo ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/libcuda.so.1
sudo ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/libcuda.so
假设/usr/local/cuda是CUDA的安装路径（不同平台可能不一样）。一旦完成，就可以在 CPU-only 实例上实际安装和使用 tensorflow-gpu。
我知道这看起来像是 hack，甚至可能无法在某些平台上运行，但至少在某种程度上可以在 GPU 和非 GPU 实例上使用相同的 requirenemts.txt 甚至相同的二进制文件。