构建 TensorFlow：从 Bazel 根外部导入 MPI headers
Building TensorFlow: import MPI headers from outside of the Bazel root
我想在 Ubuntu 16.04 上构建 TensorFlow 1.3（不是 1.13），支持 MPI（而不是默认的 gRPC）。我从 Ubuntu 仓库安装了包 libopenmpi-dev。当 运行 configure 脚本时，我提供了 /usr/lib/openmpi 作为 MPI 工具包目录。
我使用这个命令启动构建：
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
但是有一个header包含问题：
- 文件
tensorflow/contrib/mpi/mpi_utils.cc 包含 tensorflow/contrib/mpi/mpi_utils.h
- 这个
mpi_utils.h包括third_party/mpi/mpi.h
- 此
mpi.h 是指向 /usr/lib/openmpi/include/mpi.h 的符号链接
- 实际
mpi.h 包含此行：
#include "openmpi/ompi/mpi/cxx/mpicxx.h"
- 并且
mpicxx.h在文件夹/usr/lib/openmpi/include/openmpi/ompi/mpi/cxx/中，不在包含路径中。
我 "fixed" 通过创建指向正确文件夹的符号链接来实现此目的：
$ ln -s /usr/lib/openmpi/include/openmpi third_party/mpi/openmpi
现在 mpicxx.h 找到了，但它想包含 mpi.h，由于 mpi.h 不在同一个文件夹中，所以失败了：
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
WARNING: /home/arno/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /home/arno/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Found 1 target...
ERROR: /home/arno/tensorflow/tensorflow/contrib/mpi/BUILD:60:1: C++ compilation of rule '//tensorflow/contrib/mpi:mpi_rendezvous_mgr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 151 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./third_party/mpi/mpi.h:2673:0,
from ./tensorflow/contrib/mpi/mpi_utils.h:27,
from ./tensorflow/contrib/mpi/mpi_rendezvous_mgr.h:33,
from tensorflow/contrib/mpi/mpi_rendezvous_mgr.cc:18:
./third_party/mpi/openmpi/ompi/mpi/cxx/mpicxx.h:35:17: fatal error: mpi.h: No such file or directory
compilation terminated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/arno/tensorflow/tensorflow/tools/pip_package/BUILD:134:1 C++ compilation of rule '//tensorflow/contrib/mpi:mpi_rendezvous_mgr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 151 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
INFO: Elapsed time: 6.668s, Critical Path: 4.98s
我尝试使用以下命令将 headers 的路径手动添加到 GCC 的包含路径中：
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --copt='-I/usr/lib/openmpi/include'
...但后来我收到错误，因为 /usr/lib/openmpi/include/openmpi/ompi/mpi/cxx 中包含的 header 未在 Bazel 的配置文件中声明。而且我无法向 Bazel 声明它们，因为它不接受绝对路径。
我找不到使此构建工作的正确方法。我是 Bazel 的新手，根据我的阅读，构建目录应该是 "self-contained"，即包含所有必要的 header 和源文件，但是 TensorFlow 存储库通过添加符号链接违反了这一点在 third_party/mpi 中 /usr/lib/...。更改 TensorFlow 版本不是一个选项。
如何构建支持 OpenMPI 的 TensorFlow 1.3？
编辑：将 -s 选项添加到 Bazel 构建命令，如评论中所建议的那样，提供了更详细的输出，但我无法辨别使用了哪个编译器。我认为这些是相关的行：
>>>>> # //tensorflow/core/kernels:transpose_functor [action 'Compiling tensorflow/core/kernels/transpose_functor_cpu.cc']
(cd /home/arno/.cache/bazel/_bazel_arno/e7d941e3336cbc1a05a122432422a066/execroot/tensorflow && \
exec env - \
CUDA_TOOLKIT_PATH=/usr/local/cuda \
CUDNN_INSTALL_PATH=/usr/local/lib \
GCC_HOST_COMPILER_PATH=/usr/bin/gcc \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/arno/bin \
PYTHON_BIN_PATH=/usr/bin/python3 \
PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
TF_CUDA_CLANG=0 \
TF_CUDA_COMPUTE_CAPABILITIES=6.2 \
TF_CUDA_VERSION=8.0 \
TF_CUDNN_VERSION=6 \
TF_NEED_CUDA=1 \
TF_NEED_OPENCL=0 \
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' -MD -MF bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/transpose_functor/tensorflow/core/kernels/transpose_functor_cpu.pic.d '-frandom-seed=bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/transpose_functor/tensorflow/core/kernels/transpose_functor_cpu.pic.o' -fPIC -DEIGEN_MPL2_ONLY -DTENSORFLOW_USE_JEMALLOC -DSNAPPY -DTENSORFLOW_USE_MPI -iquote . -iquote bazel-out/local_linux-py3-opt/genfiles -iquote external/jemalloc -iquote bazel-out/local_linux-py3-opt/genfiles/external/jemalloc -iquote external/bazel_tools -iquote bazel-out/local_linux-py3-opt/genfiles/external/bazel_tools -iquote external/protobuf -iquote bazel-out/local_linux-py3-opt/genfiles/external/protobuf -iquote external/eigen_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local_linux-py3-opt/genfiles/external/local_config_sycl -iquote external/gif_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/gif_archive -iquote external/jpeg -iquote bazel-out/local_linux-py3-opt/genfiles/external/jpeg -iquote external/com_googlesource_code_re2 -iquote bazel-out/local_linux-py3-opt/genfiles/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/local_linux-py3-opt/genfiles/external/fft2d -iquote external/highwayhash -iquote bazel-out/local_linux-py3-opt/genfiles/external/highwayhash -iquote external/png_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/png_archive -iquote external/zlib_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/zlib_archive -iquote external/snappy -iquote bazel-out/local_linux-py3-opt/genfiles/external/snappy -iquote external/local_config_cuda -iquote bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda -isystem external/jemalloc/include -isystem bazel-out/local_linux-py3-opt/genfiles/external/jemalloc/include -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/protobuf/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/protobuf/src -isystem external/eigen_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/eigen_archive -isystem external/gif_archive/lib -isystem bazel-out/local_linux-py3-opt/genfiles/external/gif_archive/lib -isystem external/farmhash_archive/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive/src -isystem external/png_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/png_archive -isystem external/zlib_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/zlib_archive -isystem external/local_config_cuda/cuda -isystem bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda/cuda/cuda/include -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-DGOOGLE_CUDA=1' -msse3 -pthread '-DGOOGLE_CUDA=1' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c tensorflow/core/kernels/transpose_functor_cpu.cc -o bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/transpose_functor/tensorflow/core/kernels/transpose_functor_cpu.pic.o)
ERROR: /home/arno/tensorflow/tensorflow/contrib/mpi/BUILD:48:1: C++ compilation of rule '//tensorflow/contrib/mpi:mpi_utils' failed: crosstool_wrapper_driver_is_n
ot_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 131 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./third_party/mpi/mpi.h:2673:0,
from ./tensorflow/contrib/mpi/mpi_utils.h:27,
from tensorflow/contrib/mpi/mpi_utils.cc:18:
./third_party/mpi/openmpi/ompi/mpi/cxx/mpicxx.h:35:17: fatal error: mpi.h: No such file or directory
compilation terminated.
解决方法是从源代码构建和安装 MVAPICH（MPI 工具包路径为 /usr/local）。该问题仅存在于 OpenMPI 中。
我想在 Ubuntu 16.04 上构建 TensorFlow 1.3（不是 1.13），支持 MPI（而不是默认的 gRPC）。我从 Ubuntu 仓库安装了包 libopenmpi-dev。当 运行 configure 脚本时，我提供了 /usr/lib/openmpi 作为 MPI 工具包目录。
我使用这个命令启动构建：
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
但是有一个header包含问题：
- 文件
tensorflow/contrib/mpi/mpi_utils.cc包含tensorflow/contrib/mpi/mpi_utils.h - 这个
mpi_utils.h包括third_party/mpi/mpi.h - 此
mpi.h是指向/usr/lib/openmpi/include/mpi.h 的符号链接
- 实际
mpi.h包含此行：
#include "openmpi/ompi/mpi/cxx/mpicxx.h"
- 并且
mpicxx.h在文件夹/usr/lib/openmpi/include/openmpi/ompi/mpi/cxx/中，不在包含路径中。
我 "fixed" 通过创建指向正确文件夹的符号链接来实现此目的：
$ ln -s /usr/lib/openmpi/include/openmpi third_party/mpi/openmpi
现在 mpicxx.h 找到了，但它想包含 mpi.h，由于 mpi.h 不在同一个文件夹中，所以失败了：
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
WARNING: /home/arno/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /home/arno/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Found 1 target...
ERROR: /home/arno/tensorflow/tensorflow/contrib/mpi/BUILD:60:1: C++ compilation of rule '//tensorflow/contrib/mpi:mpi_rendezvous_mgr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 151 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./third_party/mpi/mpi.h:2673:0,
from ./tensorflow/contrib/mpi/mpi_utils.h:27,
from ./tensorflow/contrib/mpi/mpi_rendezvous_mgr.h:33,
from tensorflow/contrib/mpi/mpi_rendezvous_mgr.cc:18:
./third_party/mpi/openmpi/ompi/mpi/cxx/mpicxx.h:35:17: fatal error: mpi.h: No such file or directory
compilation terminated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/arno/tensorflow/tensorflow/tools/pip_package/BUILD:134:1 C++ compilation of rule '//tensorflow/contrib/mpi:mpi_rendezvous_mgr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 151 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
INFO: Elapsed time: 6.668s, Critical Path: 4.98s
我尝试使用以下命令将 headers 的路径手动添加到 GCC 的包含路径中：
$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --copt='-I/usr/lib/openmpi/include'
...但后来我收到错误，因为 /usr/lib/openmpi/include/openmpi/ompi/mpi/cxx 中包含的 header 未在 Bazel 的配置文件中声明。而且我无法向 Bazel 声明它们，因为它不接受绝对路径。
我找不到使此构建工作的正确方法。我是 Bazel 的新手，根据我的阅读，构建目录应该是 "self-contained"，即包含所有必要的 header 和源文件，但是 TensorFlow 存储库通过添加符号链接违反了这一点在 third_party/mpi 中 /usr/lib/...。更改 TensorFlow 版本不是一个选项。
如何构建支持 OpenMPI 的 TensorFlow 1.3？
编辑：将 -s 选项添加到 Bazel 构建命令，如评论中所建议的那样，提供了更详细的输出，但我无法辨别使用了哪个编译器。我认为这些是相关的行：
>>>>> # //tensorflow/core/kernels:transpose_functor [action 'Compiling tensorflow/core/kernels/transpose_functor_cpu.cc']
(cd /home/arno/.cache/bazel/_bazel_arno/e7d941e3336cbc1a05a122432422a066/execroot/tensorflow && \
exec env - \
CUDA_TOOLKIT_PATH=/usr/local/cuda \
CUDNN_INSTALL_PATH=/usr/local/lib \
GCC_HOST_COMPILER_PATH=/usr/bin/gcc \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/arno/bin \
PYTHON_BIN_PATH=/usr/bin/python3 \
PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
TF_CUDA_CLANG=0 \
TF_CUDA_COMPUTE_CAPABILITIES=6.2 \
TF_CUDA_VERSION=8.0 \
TF_CUDNN_VERSION=6 \
TF_NEED_CUDA=1 \
TF_NEED_OPENCL=0 \
external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' -MD -MF bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/transpose_functor/tensorflow/core/kernels/transpose_functor_cpu.pic.d '-frandom-seed=bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/transpose_functor/tensorflow/core/kernels/transpose_functor_cpu.pic.o' -fPIC -DEIGEN_MPL2_ONLY -DTENSORFLOW_USE_JEMALLOC -DSNAPPY -DTENSORFLOW_USE_MPI -iquote . -iquote bazel-out/local_linux-py3-opt/genfiles -iquote external/jemalloc -iquote bazel-out/local_linux-py3-opt/genfiles/external/jemalloc -iquote external/bazel_tools -iquote bazel-out/local_linux-py3-opt/genfiles/external/bazel_tools -iquote external/protobuf -iquote bazel-out/local_linux-py3-opt/genfiles/external/protobuf -iquote external/eigen_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/eigen_archive -iquote external/local_config_sycl -iquote bazel-out/local_linux-py3-opt/genfiles/external/local_config_sycl -iquote external/gif_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/gif_archive -iquote external/jpeg -iquote bazel-out/local_linux-py3-opt/genfiles/external/jpeg -iquote external/com_googlesource_code_re2 -iquote bazel-out/local_linux-py3-opt/genfiles/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/local_linux-py3-opt/genfiles/external/fft2d -iquote external/highwayhash -iquote bazel-out/local_linux-py3-opt/genfiles/external/highwayhash -iquote external/png_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/png_archive -iquote external/zlib_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/zlib_archive -iquote external/snappy -iquote bazel-out/local_linux-py3-opt/genfiles/external/snappy -iquote external/local_config_cuda -iquote bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda -isystem external/jemalloc/include -isystem bazel-out/local_linux-py3-opt/genfiles/external/jemalloc/include -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/protobuf/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/protobuf/src -isystem external/eigen_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/eigen_archive -isystem external/gif_archive/lib -isystem bazel-out/local_linux-py3-opt/genfiles/external/gif_archive/lib -isystem external/farmhash_archive/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive/src -isystem external/png_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/png_archive -isystem external/zlib_archive -isystem bazel-out/local_linux-py3-opt/genfiles/external/zlib_archive -isystem external/local_config_cuda/cuda -isystem bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/local_linux-py3-opt/genfiles/external/local_config_cuda/cuda/cuda/include -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-DGOOGLE_CUDA=1' -msse3 -pthread '-DGOOGLE_CUDA=1' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c tensorflow/core/kernels/transpose_functor_cpu.cc -o bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/transpose_functor/tensorflow/core/kernels/transpose_functor_cpu.pic.o)
ERROR: /home/arno/tensorflow/tensorflow/contrib/mpi/BUILD:48:1: C++ compilation of rule '//tensorflow/contrib/mpi:mpi_utils' failed: crosstool_wrapper_driver_is_n
ot_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 131 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
In file included from ./third_party/mpi/mpi.h:2673:0,
from ./tensorflow/contrib/mpi/mpi_utils.h:27,
from tensorflow/contrib/mpi/mpi_utils.cc:18:
./third_party/mpi/openmpi/ompi/mpi/cxx/mpicxx.h:35:17: fatal error: mpi.h: No such file or directory
compilation terminated.
解决方法是从源代码构建和安装 MVAPICH（MPI 工具包路径为 /usr/local）。该问题仅存在于 OpenMPI 中。