Pybind11 共享库可见性问题
Pybind11 Shared Library Visibility Issues
我们正在尝试使用 pybind11 将一些 c++ 代码编译到共享库中。此共享库不调用 python 函数，而是从 python 脚本调用。然后我们要创建一个入口点 C++ 可执行文件，它使用这个共享库来测试 C++ 代码。它不以任何方式与 pybind 或 python 端交互。
当我们这样做时，我们的共享库编译成功。但是，在可执行目标中，我们会遇到各种链接器错误，这些链接器错误指定 'undefined reference to' 我们在共享库中的构造函数。
阅读 pybind 文档后，我们认为这是因为 pybind11_add_module 函数强制 cmake 使用 'hidden' 可见性标志编译共享库。我们随后尝试在我们的 c++ 共享库中公开相关函数，以便我们的可执行文件可以访问它（使用 __attribute__((visibility("default")))）。这解决了对我们的构造函数的未定义引用，但会导致下面提供的错误。请注意，这些错误发生在编译我们的可执行文件时。共享库仍然编译成功。
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyTuple_SetItem'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_Repr'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyInstanceMethod_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_ValueError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyLong_FromSsize_t'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyDict_GetItemString'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_Py_TrueStruct'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_IndexError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_NormalizeException'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyInstanceMethod_New'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyEval_AcquireThread'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_Str'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyThreadState_DeleteCurrent'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyGILState_GetThisThreadState'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_GetAttrString'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCapsule_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyModule_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyMem_Free'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_Restore'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyType_IsSubtype'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyModule_AddObject'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_WarnEx'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCapsule_SetPointer'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyTuple_New'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_SetAttr'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_IsInstance'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyEval_RestoreThread'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_Py_NoneStruct'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyException_SetTraceback'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyUnicode_FromFormat'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_Append'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_MemoryError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyType_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyDict_Next'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_Size'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyTuple_Size'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyBuffer_Release'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_Format'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_CallObject'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyFloat_FromDouble'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyUnicode_DecodeUTF8'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_Py_Dealloc'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCFunction_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_OverflowError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCFunction_NewEx'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_New'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyProperty_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_PyObject_GetDictPtr'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyUnicode_FromString'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `Py_GetVersion'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCapsule_SetContext'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyFrame_GetLineNumber'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyThread_tss_get'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyBytes_Size'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PySequence_Check'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_GetItem'
... [additional reference errors removed for brevity]
[build] collect2: error: ld returned 1 exit status
我们已经尝试了各种关于共享库编译可见性的命令，但总是以上述一个或两个错误告终。在下面找到我们的 CMAKEFILES：
编译共享库的主要 CMakeLists.txt（为简洁起见删除了 gstreamer、cuda 等不相关的命令...）：
cmake_minimum_required(VERSION 3.5)
project(visual_processing_node)
# Default to C99
if(NOT CMAKE_C_STANDARD)
set(CMAKE_C_STANDARD 99)
endif()
# Default to C++14
if(NOT CMAKE_CXX_STANDARD)
set(CMAKE_CXX_STANDARD 14)
endif()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread")
find_package(Threads REQUIRED)
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
add_compile_options(-Wall -Wextra -Wpedantic)
endif()
# Fetch pybind11
include(FetchContent)
FetchContent_Declare(
pybind11
GIT_REPOSITORY https://github.com/pybind/pybind11.git
GIT_TAG v2.6.2
GIT_SHALLOW TRUE
)
FetchContent_MakeAvailable(pybind11)
pybind11_add_module(pipeline_manager SHARED src/deepstream_pipeline.cpp src/deepstream_pipeline_helpers.cpp)
target_include_directories(
pipeline_manager
PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
${GSTREAMER_INCLUDE_DIRS} ${GSTREAMER_APP_INCLUDE_DIRS}
${GLIB_INCLUDE_DIRS} ${GLOG_INCLUDE_DIRS} "${DeepStream_DIR}/lib")
target_link_libraries(
pipeline_manager
PUBLIC
${GSTREAMER_LIBRARIES}
${GLIB_LIBRARIES}
${GLIB_GIO_LIBRARIES}
${GLIB_GOBJECT_LIBRARIES}
${GLOG_LIBRARIES}
${GSTREAMER_APP_LIBRARIES}
"/opt/nvidia/deepstream/deepstream-6.0/lib/libnvdsgst_meta.so"
"/opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_meta.so"
gstrtspserver-1.0
Threads::Threads)
add_subdirectory (tests)
'tests' 目录中的第二个 CMakeLists.txt 文件，用于编译可执行文件。
add_executable (inferenceTest inference_test.cpp)
target_link_libraries (inferenceTest pipeline_manager
pybind11::module)
target_include_directories(inferenceTest PUBLIC
"${PROJECT_BINARY_DIR}")
我不确定是什么原因导致您收到错误，但我可以建议一个解决方法，也许是更好的方法。
您可以构建两个共享库：一个包含您需要测试的 C++ 代码，以及一个动态 link 到第一个共享库并且仅提供绑定定义的 Pybind11 扩展。
然后你可以 link 第一个用于测试，而不 Python 妨碍。
我们正在尝试使用 pybind11 将一些 c++ 代码编译到共享库中。此共享库不调用 python 函数，而是从 python 脚本调用。然后我们要创建一个入口点 C++ 可执行文件，它使用这个共享库来测试 C++ 代码。它不以任何方式与 pybind 或 python 端交互。
当我们这样做时，我们的共享库编译成功。但是，在可执行目标中，我们会遇到各种链接器错误，这些链接器错误指定 'undefined reference to' 我们在共享库中的构造函数。
阅读 pybind 文档后，我们认为这是因为 pybind11_add_module 函数强制 cmake 使用 'hidden' 可见性标志编译共享库。我们随后尝试在我们的 c++ 共享库中公开相关函数，以便我们的可执行文件可以访问它（使用 __attribute__((visibility("default")))）。这解决了对我们的构造函数的未定义引用，但会导致下面提供的错误。请注意，这些错误发生在编译我们的可执行文件时。共享库仍然编译成功。
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyTuple_SetItem'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_Repr'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyInstanceMethod_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_ValueError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyLong_FromSsize_t'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyDict_GetItemString'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_Py_TrueStruct'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_IndexError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_NormalizeException'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyInstanceMethod_New'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyEval_AcquireThread'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_Str'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyThreadState_DeleteCurrent'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyGILState_GetThisThreadState'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_GetAttrString'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCapsule_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyModule_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyMem_Free'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_Restore'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyType_IsSubtype'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyModule_AddObject'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_WarnEx'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCapsule_SetPointer'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyTuple_New'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_SetAttr'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_IsInstance'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyEval_RestoreThread'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_Py_NoneStruct'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyException_SetTraceback'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyUnicode_FromFormat'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_Append'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_MemoryError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyType_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyDict_Next'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_Size'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyTuple_Size'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyBuffer_Release'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyErr_Format'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyObject_CallObject'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyFloat_FromDouble'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyUnicode_DecodeUTF8'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_Py_Dealloc'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCFunction_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyExc_OverflowError'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCFunction_NewEx'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_New'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyProperty_Type'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `_PyObject_GetDictPtr'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyUnicode_FromString'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `Py_GetVersion'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyCapsule_SetContext'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyFrame_GetLineNumber'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyThread_tss_get'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyBytes_Size'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PySequence_Check'
[build] ../pipeline_manager.cpython-38-x86_64-linux-gnu.so: undefined reference to `PyList_GetItem'
... [additional reference errors removed for brevity]
[build] collect2: error: ld returned 1 exit status
我们已经尝试了各种关于共享库编译可见性的命令，但总是以上述一个或两个错误告终。在下面找到我们的 CMAKEFILES：
编译共享库的主要 CMakeLists.txt（为简洁起见删除了 gstreamer、cuda 等不相关的命令...）：
cmake_minimum_required(VERSION 3.5)
project(visual_processing_node)
# Default to C99
if(NOT CMAKE_C_STANDARD)
set(CMAKE_C_STANDARD 99)
endif()
# Default to C++14
if(NOT CMAKE_CXX_STANDARD)
set(CMAKE_CXX_STANDARD 14)
endif()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread")
find_package(Threads REQUIRED)
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
add_compile_options(-Wall -Wextra -Wpedantic)
endif()
# Fetch pybind11
include(FetchContent)
FetchContent_Declare(
pybind11
GIT_REPOSITORY https://github.com/pybind/pybind11.git
GIT_TAG v2.6.2
GIT_SHALLOW TRUE
)
FetchContent_MakeAvailable(pybind11)
pybind11_add_module(pipeline_manager SHARED src/deepstream_pipeline.cpp src/deepstream_pipeline_helpers.cpp)
target_include_directories(
pipeline_manager
PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>
${GSTREAMER_INCLUDE_DIRS} ${GSTREAMER_APP_INCLUDE_DIRS}
${GLIB_INCLUDE_DIRS} ${GLOG_INCLUDE_DIRS} "${DeepStream_DIR}/lib")
target_link_libraries(
pipeline_manager
PUBLIC
${GSTREAMER_LIBRARIES}
${GLIB_LIBRARIES}
${GLIB_GIO_LIBRARIES}
${GLIB_GOBJECT_LIBRARIES}
${GLOG_LIBRARIES}
${GSTREAMER_APP_LIBRARIES}
"/opt/nvidia/deepstream/deepstream-6.0/lib/libnvdsgst_meta.so"
"/opt/nvidia/deepstream/deepstream-6.0/lib/libnvds_meta.so"
gstrtspserver-1.0
Threads::Threads)
add_subdirectory (tests)
'tests' 目录中的第二个 CMakeLists.txt 文件，用于编译可执行文件。
add_executable (inferenceTest inference_test.cpp)
target_link_libraries (inferenceTest pipeline_manager
pybind11::module)
target_include_directories(inferenceTest PUBLIC
"${PROJECT_BINARY_DIR}")
我不确定是什么原因导致您收到错误，但我可以建议一个解决方法，也许是更好的方法。
您可以构建两个共享库：一个包含您需要测试的 C++ 代码，以及一个动态 link 到第一个共享库并且仅提供绑定定义的 Pybind11 扩展。
然后你可以 link 第一个用于测试，而不 Python 妨碍。