如何构建和分发依赖于第三方 libFoo.so 的 Python/Cython 包

How to build and distribute a Python/Cython package that depends on third party libFoo.so

我写了一个 Python 模块,它依赖于一些 C 扩展。这些 C 扩展依次依赖于几个已编译的 C 库。我希望能够分发与所有依赖项捆绑在一起的此模块。

我整理了一个最小的例子 (it can be found on GitHub in its entirety)。

目录结构为:

$ tree .
.
├── README.md
├── poc
│   ├── __init__.py
│   ├── cython_extensions
│   │   ├── __init__.py
│   │   ├── cvRoberts_dns.c
│   │   ├── cvRoberts_dns.h
│   │   ├── helloworld.c
│   │   ├── helloworld.pxd
│   │   ├── helloworld.pyx
│   │   ├── test.c
│   │   └── test.h
│   ├── do_stuff.c
│   └── do_stuff.pyx
└── setup.py

setup.py 构建扩展,并链接必要的库(在本例中为 libsundials_cvodelibsundials_nvectorserial):

from setuptools import setup, find_packages
from setuptools.extension import Extension
from Cython.Build import cythonize


ext_module_dostuff = Extension(
    'poc.do_stuff',
    ['poc/do_stuff.pyx'],
)

ext_module_helloworld = Extension(
    'poc.cython_extensions.helloworld',
    ['poc/cython_extensions/helloworld.pyx', 'poc/cython_extensions/test.c', 'poc/cython_extensions/cvRoberts_dns.c'],
    include_dirs = ['/usr/local/include'],
    libraries = ['m', 'sundials_cvodes', 'sundials_nvecserial'],
    library_dirs = ['/usr/local/lib'],
)

cython_ext_modules = [
   ext_module_dostuff,
   ext_module_helloworld
]


setup (
  name = "poc",
  ext_modules = cythonize(cython_ext_modules),
  packages=['poc', 'poc.cython_extensions'],
)

这一切都很好,但它确实需要最终用户首先安装日晷(而且,在实际情况下,还有其他几个非常挑剔的库 运行)。

理想情况下,我希望能够仅在开发机器上进行设置,创建一个包含适当共享库的发行版,并发布某种捆绑包。

考虑到到目前为止我找到的各种教程、示例和 SO 帖子。我被引导相信我走在正确的轨道上。然而,还有一些最后的步骤我只是不摸不着头脑。

感谢任何帮助:-)。

您可能知道,分发带有已编译组件的 Python 模块的推荐方法是使用 wheel format。似乎没有任何标准的跨平台方式可以将第三方本机库捆绑到一个轮子中。但是,有用于此目的的特定于平台的工具。

在 Linux 上,使用 auditwheel

auditwheel modifies an existing Linux wheel file to add any third-party libraries which are not included in the basic "manylinux" 标准。以下是如何在 Ubuntu 17.10 的全新安装中将它与您的项目一起使用的演练:

首先,安装基本的Python开发工具,以及带有头文件的第三方库:

root@ubuntu-17:~# apt-get install cython python-pip unzip
root@ubuntu-17:~# apt-get install libsundials-serial-dev

然后将您的项目构建到 wheel 文件中:

root@ubuntu-17:~# cd cython-example/
root@ubuntu-17:~/cython-example# python setup.py bdist_wheel
[...]
root@ubuntu-17:~/cython-example# cd dist/
root@ubuntu-17:~/cython-example/dist# ll
total 80
drwxr-xr-x 2 root root  4096 Nov  8 11:28 ./
drwxr-xr-x 7 root root  4096 Nov  8 11:28 ../
-rw-r--r-- 1 root root 70135 Nov  8 11:28 poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Archive:  poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   116648  2017-11-08 11:28   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
      105  2017-11-08 11:28   poc-0.0.0.dist-info/WHEEL
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
      793  2017-11-08 11:28   poc-0.0.0.dist-info/RECORD
---------                     -------
   180382                     10 files

wheel文件现在可以在本地安装并测试:

root@ubuntu-17:~/cython-example/dist# pip install poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

现在我们安装 auditwheel 工具。它需要 Python 3,但它能够处理 Python 2 或 3 的轮子。

root@ubuntu-17:~/cython-example/dist# apt-get install python3-pip
root@ubuntu-17:~/cython-example/dist# pip3 install auditwheel

auditwheel 使用另一个名为 patchelf 的工具来完成它的工作。不幸的是,缺少 Ubuntu 17.10 中包含的 patchelf 版本 a bugfix without which auditwheel will not work. So we'll have to build it from source (script taken from the manylinux Docker image):

root@ubuntu-17:~# apt-get install autoconf
root@ubuntu-17:~# PATCHELF_VERSION=6bfcafbba8d89e44f9ac9582493b4f27d9d8c369
root@ubuntu-17:~# curl -sL -o patchelf.tar.gz https://github.com/NixOS/patchelf/archive/$PATCHELF_VERSION.tar.gz
root@ubuntu-17:~# tar -xzf patchelf.tar.gz
root@ubuntu-17:~# (cd patchelf-$PATCHELF_VERSION && ./bootstrap.sh && ./configure && make && make install)

现在我们可以查看wheel需要哪些第三方库了:

root@ubuntu-17:~/cython-example/dist# auditwheel show poc-0.0.0-cp27-cp27mu-linux_x86_64.whl

poc-0.0.0-cp27-cp27mu-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.4',
'GLIBC_2.2.5', 'GLIBC_2.3.4'}

The following external shared libraries are required by the wheel:
{
    "libblas.so.3": "/usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1",
    "libc.so.6": "/lib/x86_64-linux-gnu/libc-2.26.so",
    "libgcc_s.so.1": "/lib/x86_64-linux-gnu/libgcc_s.so.1",
    "libgfortran.so.4": "/usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0",
    "liblapack.so.3": "/usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1",
    "libm.so.6": "/lib/x86_64-linux-gnu/libm-2.26.so",
    "libpthread.so.0": "/lib/x86_64-linux-gnu/libpthread-2.26.so",
    "libquadmath.so.0": "/usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0",
    "libsundials_cvodes.so.2": "/usr/lib/libsundials_cvodes.so.2.0.0",
    "libsundials_nvecserial.so.0": "/usr/lib/libsundials_nvecserial.so.0.0.2"
}

In order to achieve the tag platform tag "manylinux1_x86_64" the
following shared library dependencies will need to be eliminated:

libblas.so.3, libgfortran.so.4, liblapack.so.3, libquadmath.so.0,
libsundials_cvodes.so.2, libsundials_nvecserial.so.0

并创建一个捆绑它们的新轮子:

root@ubuntu-17:~/cython-example/dist# auditwheel repair poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Repairing poc-0.0.0-cp27-cp27mu-linux_x86_64.whl
Grafting: /usr/lib/libsundials_nvecserial.so.0.0.2 -> poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
Grafting: /usr/lib/libsundials_cvodes.so.2.0.0 -> poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
Grafting: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1 -> poc/.libs/liblapack-549933c4.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1 -> poc/.libs/libblas-52fa99c8.so.3.7.1
Grafting: /usr/lib/x86_64-linux-gnu/libgfortran.so.4.0.0 -> poc/.libs/libgfortran-2df4b07d.so.4.0.0
Grafting: /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0 -> poc/.libs/libquadmath-0d7c3070.so.0.0.0
Setting RPATH: poc/cython_extensions/helloworld.so to "$ORIGIN/../.libs"
Previous filename tags: linux_x86_64
New filename tags: manylinux1_x86_64
Previous WHEEL info tags: cp27-cp27mu-linux_x86_64
New WHEEL info tags: cp27-cp27mu-manylinux1_x86_64

Fixed-up wheel written to /root/cython-example/dist/wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
root@ubuntu-17:~/cython-example/dist# unzip -l wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
Archive:  wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
      167  2017-11-08 11:28   poc-0.0.0.dist-info/METADATA
        4  2017-11-08 11:28   poc-0.0.0.dist-info/top_level.txt
       10  2017-11-08 11:28   poc-0.0.0.dist-info/DESCRIPTION.rst
      211  2017-11-08 11:28   poc-0.0.0.dist-info/metadata.json
     1400  2017-11-08 12:08   poc-0.0.0.dist-info/RECORD
      110  2017-11-08 12:08   poc-0.0.0.dist-info/WHEEL
    62440  2017-11-08 11:28   poc/do_stuff.so
        2  2017-11-08 11:28   poc/__init__.py
   131712  2017-11-08 12:08   poc/cython_extensions/helloworld.so
        2  2017-11-08 11:28   poc/cython_extensions/__init__.py
   230744  2017-11-08 12:08   poc/.libs/libsundials_cvodes-50fde5ee.so.2.0.0
  7005072  2017-11-08 12:08   poc/.libs/liblapack-549933c4.so.3.7.1
   264024  2017-11-08 12:08   poc/.libs/libquadmath-0d7c3070.so.0.0.0
  2039960  2017-11-08 12:08   poc/.libs/libgfortran-2df4b07d.so.4.0.0
    17736  2017-11-08 12:08   poc/.libs/libsundials_nvecserial-42b4120e.so.0.0.2
   452432  2017-11-08 12:08   poc/.libs/libblas-52fa99c8.so.3.7.1
---------                     -------
 10206026                     16 files

如果我们卸载第三方库,之前安装的轮子将停止工作:

root@ubuntu-17:~/cython-example/dist# apt-get remove libsundials-serial-dev && apt-get autoremove
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "poc/do_stuff.pyx", line 1, in init poc.do_stuff
ImportError: libsundials_cvodes.so.2: cannot open shared object file: No such file or directory

但是带有捆绑库的轮子可以正常工作:

root@ubuntu-17:~/cython-example/dist# pip uninstall poc
[...]
root@ubuntu-17:~/cython-example/dist# pip install wheelhouse/poc-0.0.0-cp27-cp27mu-manylinux1_x86_64.whl
[...]
root@ubuntu-17:~/cython-example/dist# python -c "from poc.do_stuff import hello; hello()"
hello cython
0.841470984808
trying to load the sundials program

3-species kinetics problem

At t = 2.6391e-01      y =  9.899653e-01    3.470564e-05    1.000000e-02
    rootsfound[] =   0   1
At t = 4.0000e-01      y =  9.851641e-01    3.386242e-05    1.480205e-02
[...]

在 OSX 上,使用 delocate

delocate for OSX 显然与 auditwheel 非常相似。不幸的是,我没有可用的 OSX 机器来提供演练。

组合示例:

一个同时使用这两种工具的项目是 SciPy。 This repository, despite its name, contains the official SciPy build process for all platforms, not just Mac. Specifically, compare the Linux build script (which uses auditwheel), with the OSX build script(使用 delocate)。

要查看此过程的结果,您可能需要下载并解压缩一些 SciPy wheels from PyPI。例如,scipy-1.0.0-cp27-cp27m-manylinux1_x86_64.whl 包含以下内容:

 38513408  2017-10-25 06:02   scipy/.libs/libopenblasp-r0-39a31c03.2.18.so
  1023960  2017-10-25 06:02   scipy/.libs/libgfortran-ed201abd.so.3.0.0

虽然 scipy-1.0.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl 包含此内容:

   273072  2017-10-25 07:03   scipy/.dylibs/libgcc_s.1.dylib
  1550456  2017-10-25 07:03   scipy/.dylibs/libgfortran.3.dylib
   279932  2017-10-25 07:03   scipy/.dylibs/libquadmath.0.dylib

我建议采用完全不同的方法。设置 Linux 包管理基础设施。在 Ubuntu/Debian 上,这可以用 reprepro 完成。 https://wiki.ubuntuusers.de/reprepro/ 可能是一个开始,但还有更多可用的教程。然后,您可以构建自己的 Linux 程序包,将您的库和所有必需的文件与您的 Python 应用程序一起分发。

对于您的客户来说,这将是一种非常简洁和方便的方法。特别是关于更新。 (您甚至可以根据需要同时处理不同的 OS 版本。)

一如既往,干净的方法是有代价的。这种干净的方法需要您付出很多努力才能实现。您不仅需要设置服务器 - 这是比较容易的部分 - 还需要了解如何构建包 - 这并不困难,但您需要阅读一些如何执行此操作并进行大量实验才能结束提供完全符合您要求的包裹。然而,一切都会如你所愿。未来的更新对您和您的客户端机器来说真的很容易。

如果你想在未来简化更新,想了解 Linux 并且将来可能对自己的包有要求,我会推荐这种方法。或者客户量大。


那是一种非常 "high level" 的方法。相比之下,非常 "low level" 的方法是以下一种:

  • 在程序启动时检查库是否存在
  • 如果不存在:终止应用程序。打印引用脚本的文本,说明如何安装必要的库。这甚至可以是 URL 下载脚本的位置,f.e。与:

bash <(curl -s http://mywebsite.com/myscript.txt)

要增强 mhsmith's excellent ,以下是在 MacOS 上使用 delocate 执行的步骤:

  1. 安装 sundials,例如使用 Homebrew:

    $ brew install sundials
    
  2. 构建包:

    $ python setup.py bdist_wheel
    
  3. auditwheel show/auditwheel repair的挂件是delocate-listdeps/delocate-wheel,所以先分析生成的wheel文件:

    $ delocate-listdeps --all dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
    /usr/lib/libSystem.B.dylib
    /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib
    /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib
    
  4. 正在修复 wheel 文件:

    $ delocate-wheel -v -w dist_fixed dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
    Fixing: dist/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
    Copied to package .dylibs directory:
      /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_cvodes.2.9.0.dylib
      /usr/local/Cellar/sundials/2.7.0_3/lib/libsundials_nvecserial.2.7.0.dylib
    

dist_fixed 目录中,您将拥有捆绑的轮子。您会注意到大小差异:

$ ls -l dist/ dist_fixed/
dist/:
total 72
-rw-r--r--  1 hoefling  wheel  36030 10 Nov 20:25 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

dist_fixed/:
total 240
-rw-r--r--  1 hoefling  wheel  120101 10 Nov 20:34 poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl

如果您列出捆绑轮的 deps,您会注意到现在捆绑了所需的库(由前缀 @loader_path 表示):

$ delocate-listdeps --all dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
/usr/lib/libSystem.B.dylib
@loader_path/../.dylibs/libsundials_cvodes.2.9.0.dylib
@loader_path/../.dylibs/libsundials_nvecserial.2.7.0.dylib

安装捆绑轮(注意捆绑库已正确安装):

$ pip install dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl 
Processing ./dist_fixed/poc-0.0.0-cp27-cp27m-macosx_10_13_intel.whl
Installing collected packages: poc
Successfully installed poc-0.0.0
$ pip show -f poc
Name: poc
Version: 0.0.0
Summary: UNKNOWN
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /Users/hoefling/.virtualenvs/Whosebug-py27/lib/python2.7/site-packages
Requires: 
Files:
  poc-0.0.0.dist-info/DESCRIPTION.rst
  poc-0.0.0.dist-info/INSTALLER
  poc-0.0.0.dist-info/METADATA
  poc-0.0.0.dist-info/RECORD
  poc-0.0.0.dist-info/WHEEL
  poc-0.0.0.dist-info/metadata.json
  poc-0.0.0.dist-info/top_level.txt
  poc/.dylibs/libsundials_cvodes.2.9.0.dylib
  poc/.dylibs/libsundials_nvecserial.2.7.0.dylib
  poc/__init__.py
  poc/__init__.pyc
  poc/cython_extensions/__init__.py
  poc/cython_extensions/__init__.pyc
  poc/cython_extensions/helloworld.so
  poc/do_stuff.so