使用 Tensorflow/Bazel 从 C++ 调用 Python 时出现链接错误
Linking errors when using Tensorflow/Bazel to call Python from C++
我正在编写一个 C++ 应用程序（基于 Tensorflow Serving），我需要在其中调用 Python 和 Numpy 函数（来自 Python.h 和 numpy.h）。
此应用程序是使用 Bazel 构建的。
所以我包括 header：
#include "tensorflow/python/lib/core/numpy.h"
其中又包括 Python.h。
此文件似乎包装了 numpy 以修复 Stack Overflow post.
中描述的问题
Tensorflow 项目中需要调用 Python.h 的任何其他 C++ 代码也不会直接包含它，而只包含 tensorflow/python/lib/core/numpy.h。在 Bazel 方面，只需将 //util/python:python_headers 添加到构建的 deps 中似乎就足够了。
我的 BUILD 文件如下所示：
...
cc_library(
name = "my_library",
srcs = ["my_library.cc"],
hdrs = ["my_library.h"],
deps = [
@org_tensorflow//util/python:python_headers",
@org_tensorflow//third_party/py/numpy:headers",
] + SOME_OTHER_DEPS + TENSORFLOW_DEPS + SUPPORTED_TENSORFLOW_OPS,
)
cc_binary(
name = "my_main",
srcs = ["my_main.cc"],
deps = [":my_library"],
)
...
使用 bazel build :my_library 构建库工作正常。
以同样的方式构建二进制文件不起作用，我收到以下错误：
bazel-out/local-fastbuild/bin/servable/_objs/my_library/servable/my:library.pic.o:my_library.cc:function tensorflow::serving::NumpyInitializer::init(): error: undefined reference to 'Py_Initialize'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::MakeArgTuple(tensorflow::(anonymous namespace)::PyCall*, _object**): error: undefined reference to 'PyList_New'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::MakeArgTuple(tensorflow::(anonymous namespace)::PyCall*, _object**): error: undefined reference to 'PyList_SetItem'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::MakeArgTuple(tensorflow::(anonymous namespace)::PyCall*, _object**): error: undefined reference to 'Py_BuildValue'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::IsSingleNone(_object*): error: undefined reference to 'PyType_IsSubtype'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::IsSingleNone(_object*): error: undefined reference to '_Py_NoneStruct'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyEval_CallObjectWithKeywords'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyErr_Occurred'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyErr_Print'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyList_Size'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyList_GetItem'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyType_IsSubtype'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::ConvertNdarrayToTensor(_object*, tensorflow::Tensor*): error: undefined reference to 'PyString_AsStringAndSize'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::ConvertTensorToNdarray(tensorflow::Tensor const&, _object**): error: undefined reference to 'PyString_FromStringAndSize'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'PyGILState_Ensure'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'PyGILState_Release'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/numpy_lib/external/org_tensorflow/tensorflow/python/lib/core/numpy.pic.o:numpy.cc:function _import_array: error: undefined reference to 'PyImport_ImportModule'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/numpy_lib/external/org_tensorflow/tensorflow/python/lib/core/numpy.pic.o:numpy.cc:function _import_array: error: undefined reference to 'PyExc_ImportError'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/numpy_lib/external/org_tensorflow/tensorflow/python/lib/core/numpy.pic.o:numpy.cc:function _import_array: error: undefined reference to 'PyErr_SetString'
所以很明显 linking 错误反对 Python.h。但是，编译任何 Tensorflow-internal 目标都可以正常工作。我觉得很奇怪，现在即使在 Tensorflow 文件中也不能 link 到 Python.h。
在花了几天时间查看 Tensorflow 及其 BUILD 文件后，我不知道如何进行这项工作。
所以现在我在这里问：Tensorflow（及其 Bazel 文件）中定义的 Python 的正确包含在哪里以及如何正确包含？
一些线索似乎在 util/python/BUILD 和 tensorflow/tensorflow.bzl 的定义中。
那里似乎有相当多的 Bazel 魔法。
您需要 link 针对 libpython 的可执行文件。(a|so)。您的编译操作正确地使用了 python.h 并将一些未定义的符号放入对象中。构建可执行文件时，linking action 需要解析所有未定义的符号，所以有 python.h 已经不够了，你需要 "python.cc".
我对tensorflow的了解很差，如果你的用例是一个已解决的问题，也许团队中的人可以进一步评论。对于 "How to provide libpython" 的一般 Bazel 解决方案，我建议 external repositories and their rules. I think this email thread 会让你快到最后。
或者您可以将库复制到您的工作区并将其作为 cc_binary 规则的源代码。
我正在编写一个 C++ 应用程序（基于 Tensorflow Serving），我需要在其中调用 Python 和 Numpy 函数（来自 Python.h 和 numpy.h）。
此应用程序是使用 Bazel 构建的。
所以我包括 header：
#include "tensorflow/python/lib/core/numpy.h"
其中又包括 Python.h。
此文件似乎包装了 numpy 以修复
Tensorflow 项目中需要调用 Python.h 的任何其他 C++ 代码也不会直接包含它，而只包含 tensorflow/python/lib/core/numpy.h。在 Bazel 方面，只需将 //util/python:python_headers 添加到构建的 deps 中似乎就足够了。
我的 BUILD 文件如下所示：
...
cc_library(
name = "my_library",
srcs = ["my_library.cc"],
hdrs = ["my_library.h"],
deps = [
@org_tensorflow//util/python:python_headers",
@org_tensorflow//third_party/py/numpy:headers",
] + SOME_OTHER_DEPS + TENSORFLOW_DEPS + SUPPORTED_TENSORFLOW_OPS,
)
cc_binary(
name = "my_main",
srcs = ["my_main.cc"],
deps = [":my_library"],
)
...
使用 bazel build :my_library 构建库工作正常。
以同样的方式构建二进制文件不起作用，我收到以下错误：
bazel-out/local-fastbuild/bin/servable/_objs/my_library/servable/my:library.pic.o:my_library.cc:function tensorflow::serving::NumpyInitializer::init(): error: undefined reference to 'Py_Initialize'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::MakeArgTuple(tensorflow::(anonymous namespace)::PyCall*, _object**): error: undefined reference to 'PyList_New'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::MakeArgTuple(tensorflow::(anonymous namespace)::PyCall*, _object**): error: undefined reference to 'PyList_SetItem'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::MakeArgTuple(tensorflow::(anonymous namespace)::PyCall*, _object**): error: undefined reference to 'Py_BuildValue'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::IsSingleNone(_object*): error: undefined reference to 'PyType_IsSubtype'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::IsSingleNone(_object*): error: undefined reference to '_Py_NoneStruct'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyEval_CallObjectWithKeywords'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyErr_Occurred'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyErr_Print'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyList_Size'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyList_GetItem'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::(anonymous namespace)::DoCallPyFunc(tensorflow::(anonymous namespace)::PyCall*): error: undefined reference to 'PyType_IsSubtype'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::ConvertNdarrayToTensor(_object*, tensorflow::Tensor*): error: undefined reference to 'PyString_AsStringAndSize'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::ConvertTensorToNdarray(tensorflow::Tensor const&, _object**): error: undefined reference to 'PyString_FromStringAndSize'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'PyGILState_Ensure'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/py_func_lib/external/org_tensorflow/tensorflow/python/lib/core/py_func.pic.o:py_func.cc:function tensorflow::PyFuncOp::Compute(tensorflow::OpKernelContext*): error: undefined reference to 'PyGILState_Release'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/numpy_lib/external/org_tensorflow/tensorflow/python/lib/core/numpy.pic.o:numpy.cc:function _import_array: error: undefined reference to 'PyImport_ImportModule'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/numpy_lib/external/org_tensorflow/tensorflow/python/lib/core/numpy.pic.o:numpy.cc:function _import_array: error: undefined reference to 'PyExc_ImportError'
bazel-out/local-fastbuild/bin/external/org_tensorflow/tensorflow/python/_objs/numpy_lib/external/org_tensorflow/tensorflow/python/lib/core/numpy.pic.o:numpy.cc:function _import_array: error: undefined reference to 'PyErr_SetString'
所以很明显 linking 错误反对 Python.h。但是，编译任何 Tensorflow-internal 目标都可以正常工作。我觉得很奇怪，现在即使在 Tensorflow 文件中也不能 link 到 Python.h。
在花了几天时间查看 Tensorflow 及其 BUILD 文件后，我不知道如何进行这项工作。
所以现在我在这里问：Tensorflow（及其 Bazel 文件）中定义的 Python 的正确包含在哪里以及如何正确包含？
一些线索似乎在 util/python/BUILD 和 tensorflow/tensorflow.bzl 的定义中。
那里似乎有相当多的 Bazel 魔法。
您需要 link 针对 libpython 的可执行文件。(a|so)。您的编译操作正确地使用了 python.h 并将一些未定义的符号放入对象中。构建可执行文件时，linking action 需要解析所有未定义的符号，所以有 python.h 已经不够了，你需要 "python.cc".
我对tensorflow的了解很差，如果你的用例是一个已解决的问题，也许团队中的人可以进一步评论。对于 "How to provide libpython" 的一般 Bazel 解决方案，我建议 external repositories and their rules. I think this email thread 会让你快到最后。
或者您可以将库复制到您的工作区并将其作为 cc_binary 规则的源代码。