嵌入式 Python 解释器的未定义符号错误

Question

最初我在使用 pybind11 嵌入 anaconda Python 解释器的大型项目中遇到了这个错误。我能够通过一个简单的最小示例将其归结并重现错误。

当我运行我的可执行文件（嵌入 python）时，我得到这个错误：

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/app/Python-3.8.2-build/lib/python3.8/struct.py", line 13, in <module>
    from _struct import *
ImportError: /app/Python-3.8.2-build/lib/python3.8/lib-dynload/_struct.cpython-38-x86_64-linux-gnu.so: undefined symbol: PyByteArray_Type

起初，我从源代码构建了 Python-3.8.2。然后我从以下 C 代码编译了一个可执行文件：

#include <Python.h>

int main(int argc, char *argv[])
{
    Py_Initialize();
    PyRun_SimpleString("import struct");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    return 0;
}

使用这个命令：

gcc -o execpy execpy.c \
-I/app/Python-3.8.2-build/include/python3.8 \
-Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 \
-L/app/Python-3.8.2-build/lib  -lcrypt -lpthread -ldl  -lutil -lm \
/app/Python-3.8.2/libpython3.8.a

然后简单地执行./execpy给出上面的错误...有什么想法吗？

编辑：在这个例子中，我想 link libpython 静态就像 python 解释器不依赖于任何 libpython.so.

编辑：_struct.*.so 似乎不依赖于 libpython linked（这与我的标准相同anaconda python 解释器):

$ ldd /app/Python-3.8.2-build/lib/python3.8/lib-dynload/_struct.cpython-38-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007fff32bf0000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f71a5634000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f71a5266000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f71a5a5c000)

我还在另一台机器上检查了我的系统 python 解释器的 _struct.*.so，它有：

    linux-vdso.so.1 =>  (0x00007ffe2b3d9000)
    libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007febe24fd000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007febe22e1000)
    libc.so.6 => /lib64/libc.so.6 (0x00007febe1f13000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007febe1d0f000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007febe1b0c000)
    libm.so.6 => /lib64/libm.so.6 (0x00007febe180a000)
    /lib64/ld-linux-x86-64.so.2 (0x00007febe2c30000)

Answer 1

我认为发生这种情况是因为 PyByteArray_Type 未在 execpy.c 中引用，因此链接器使用其默认的 --gc-sections 逻辑删除未使用的符号。尝试添加以下选项之一：

-Wl,--no-gc-sections:

Enable garbage collection of unused input sections. It is ignored on targets that do not support this option. The default behaviour (of not performing this garbage collection) can be restored by specifying --no-gc-sections on the command line. Note that garbage collection for COFF and PE format targets is supported, but the implementation is currently considered to be experimental.

-Wl,--gc-keep-exported:

When --gc-sections is enabled, this option prevents garbage collection of unused input sections that contain global symbols having default or protected visibility. This option is intended to be used for executables where unreferenced sections would otherwise be garbage collected regardless of the external visibility of contained symbols. Note that this option has no effect when linking shared objects since it is already the default behaviour. This option is only supported for ELF format targets.

Answer 2

静态链接 libpython

简短回答：将 -rdynamic 添加到标志中使其适用于我。

-rdynamic 标志的文档：

-rdynamic
    Pass the flag -export-dynamic to the ELF linker, on targets that support it. This 
    instructs the linker to add all symbols, not only used ones, to the dynamic symbol 
    table. This option is needed for some uses of dlopen or to allow obtaining 
    backtraces from within a program.

动态链接 libpython

我还发现：如果要动态嵌入Python 3.8解释器（libpython3.8.so），有一些changes since version 3.8 :

On Unix, C extensions are no longer linked to libpython except on Android and Cygwin. When Python is embedded, libpython must not be loaded with RTLD_LOCAL, but RTLD_GLOBAL instead. Previously, using RTLD_LOCAL, it was already not possible to load C extensions which were not linked to libpython, like C extensions of the standard library built by the shared section of Modules/Setup. (Contributed by Victor Stinner in bpo-21536.)

另请注意（参见 here）：

To embed Python into an application, a new --embed option must be passed to python3-config --libs --embed to get -lpython3.8 (link the application to libpython). To support both 3.8 and older, try python3-config --libs --embed first and fallback to python3-config --libs (without --embed) if the previous command fails.

Add a pkg-config python-3.8-embed module to embed Python into an application: pkg-config python-3.8-embed --libs includes -lpython3.8. To support both 3.8 and older, try pkg-config python-X.Y-embed --libs first and fallback to pkg-config python-X.Y --libs (without --embed) if the previous command fails (replace X.Y with the Python version).

所以像这样动态编译和链接现在对我也有用：

gcc -o execpy execpy.c -I/app/Python-3.8.2-build/include/python3.8 \
    -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 \
    -lcrypt -lpthread -ldl  -lutil -lm -lpython3.8\
    -L/app/Python-3.8.2-build/lib/ -Wl,-rpath,/app/Python-3.8.2-build/lib/

嵌入式 Python 解释器的未定义符号错误

Undefined Symbol Error with embedded Python interpreter

python

c++

shared-libraries

dynamic-linking

静态链接 libpython

动态链接 libpython