使 virtualenv 中的 PyPy 使用来自 OS 的共享库而不是它自己的副本 (libsqlite3)
Make PyPy in virtualenv use shared library from OS rather than its own copy (libsqlite3)
在基于 Python/Django 的 open source project 中，我正在努力为 CI 使用 Travis 和 GH Actions。我们支持 PyPy，因此 运行 我们也对 PyPy 进行了 CI 测试。几个月以来，我们无法再 运行 那些 PyPy 测试成功了，因为我们一直遇到这个错误：OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name，如果我们 运行 Django 的 manage.py test 命令就会发生（在 post 末尾回溯）。 Django 中一些与 GIS 相关的功能需要 GDAL 库，而 GDAL 库又需要 SQLite3 库。而且它似乎需要在启用列元数据的情况下编译 sqlite3。
只有在没有 SQLITE_ENABLE_COLUMN_METADATA 的情况下编译安装的 sqlite3 库时，这才可以在本地重现。在 CI 服务器上搜索所有安装的 sqlite3 库后，很明显 PyPy 有自己安装的 libsqlite3.so.0 副本，它显然比 OS 安装版本更喜欢运行时间，即使 ldd 会引用 OS 库：
$ ldd -d /usr/lib/libgdal.so | grep sqlite3
libsqlite3.so.0 => /usr/local/lib/libsqlite3.so.0 (0x00007f09d6124000)
我怀疑 PyPy 在 运行 期间使用不同的库（它自己的副本）的原因是它的动态加载器（在回溯中引用）及其外观。我怀疑它自己的 libsqlite3.so.0 副本是在 CI 提供商安装 PyPy 时创建的。
我发现解决这个问题的唯一方法是在 运行 PyPy 测试之前显式预加载 OS 关卡库：
export LD_PRELOAD=/usr/local/lib/libsqlite3.so.0
然而，这感觉像是一个 hack，我想知道是否有更好的方法来做到这一点？我可以让 PyPy（或 virtualenv）只使用 OS 库而不是自己的副本，或者至少让它更新它的副本吗？
undefined symbol error 的回溯：
$ coverage run manage.py test catmaid.tests
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 377, in execute
django.setup()
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/registry.py", line 114, in populate
app_config.import_models()
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/config.py", line 211, in import_models
self.models_module = import_module(models_module_name)
File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
File "<frozen importlib._bootstrap>", line 980, in _find_and_load
File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/models.py", line 2, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
class AbstractBaseUser(models.Model):
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 121, in __new__
new_class.add_to_class('_meta', Options(meta, app_label))
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 325, in add_to_class
value.contribute_to_class(cls, name)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/options.py", line 208, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/__init__.py", line 28, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 207, in __getitem__
backend = load_backend(db['ENGINE'])
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 111, in load_backend
return import_module('%s.base' % backend_name)
File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
File "<frozen importlib._bootstrap>", line 980, in _find_and_load
File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/travis/build/[secure]/CATMAID/django/lib/custom_postgresql_psycopg2/base.py", line 33, in <module>
from django.contrib.gis.db.backends.postgis.base import DatabaseWrapper as PostGISDatabaseWrapper
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
from .features import DatabaseFeatures
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
from django.contrib.gis.db.models import aggregates
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
import django.contrib.gis.db.models.functions # NOQA
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/functions.py", line 3, in <module>
from django.contrib.gis.db.models.fields import BaseSpatialField, GeometryField
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
from django.contrib.gis import forms, gdal
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
from .fields import ( # NOQA
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/fields.py", line 2, in <module>
from django.contrib.gis.gdal import GDALException
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
from django.contrib.gis.gdal.datasource import DataSource
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
from django.contrib.gis.gdal.driver import Driver
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/libgdal.py", line 46, in <module>
lgdal = CDLL(lib_path)
File "/opt/python/pypy3.6-7.3.1/lib-python/3/ctypes/__init__.py", line 350, in __init__
pypy_dll = _ffi.CDLL(name, mode)
OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name
PyPy 不与 libsqlite3 链接，但它包含基于 CFFI 的纯Python 模块lib_pypy/_sqlite3.py。尝试删除（或重命名）PyPy 自带的 libsqlite3.so 版本；这可能就足够了。如果不是，请继续阅读。
来自 lib_pypy/_sqlite3.py 的纯 Python 逻辑导入 _sqlite3_cffi.pypy-??.so，它由 CFFI 通过执行 _sqlite3_build.py 生成。 （CFFI 也包含在 PyPy 中。）所以以 root 身份重新 运行 pypy _sqlite3_build.py 应该使它重新生成 _sqlite3_cffi.*.so，使用系统提供的 libsqlite3.so 和 sqlite3.h.
在基于 Python/Django 的 open source project 中，我正在努力为 CI 使用 Travis 和 GH Actions。我们支持 PyPy，因此 运行 我们也对 PyPy 进行了 CI 测试。几个月以来，我们无法再 运行 那些 PyPy 测试成功了，因为我们一直遇到这个错误：OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name，如果我们 运行 Django 的 manage.py test 命令就会发生（在 post 末尾回溯）。 Django 中一些与 GIS 相关的功能需要 GDAL 库，而 GDAL 库又需要 SQLite3 库。而且它似乎需要在启用列元数据的情况下编译 sqlite3。
只有在没有 SQLITE_ENABLE_COLUMN_METADATA 的情况下编译安装的 sqlite3 库时，这才可以在本地重现。在 CI 服务器上搜索所有安装的 sqlite3 库后，很明显 PyPy 有自己安装的 libsqlite3.so.0 副本，它显然比 OS 安装版本更喜欢运行时间，即使 ldd 会引用 OS 库：
$ ldd -d /usr/lib/libgdal.so | grep sqlite3
libsqlite3.so.0 => /usr/local/lib/libsqlite3.so.0 (0x00007f09d6124000)
我怀疑 PyPy 在 运行 期间使用不同的库（它自己的副本）的原因是它的动态加载器（在回溯中引用）及其外观。我怀疑它自己的 libsqlite3.so.0 副本是在 CI 提供商安装 PyPy 时创建的。
我发现解决这个问题的唯一方法是在 运行 PyPy 测试之前显式预加载 OS 关卡库：
export LD_PRELOAD=/usr/local/lib/libsqlite3.so.0
然而，这感觉像是一个 hack，我想知道是否有更好的方法来做到这一点？我可以让 PyPy（或 virtualenv）只使用 OS 库而不是自己的副本，或者至少让它更新它的副本吗？
undefined symbol error 的回溯：
$ coverage run manage.py test catmaid.tests
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
utility.execute()
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/core/management/__init__.py", line 377, in execute
django.setup()
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/__init__.py", line 24, in setup
apps.populate(settings.INSTALLED_APPS)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/registry.py", line 114, in populate
app_config.import_models()
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/apps/config.py", line 211, in import_models
self.models_module = import_module(models_module_name)
File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
File "<frozen importlib._bootstrap>", line 980, in _find_and_load
File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/models.py", line 2, in <module>
from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/auth/base_user.py", line 47, in <module>
class AbstractBaseUser(models.Model):
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 121, in __new__
new_class.add_to_class('_meta', Options(meta, app_label))
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/base.py", line 325, in add_to_class
value.contribute_to_class(cls, name)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/models/options.py", line 208, in contribute_to_class
self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/__init__.py", line 28, in __getattr__
return getattr(connections[DEFAULT_DB_ALIAS], item)
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 207, in __getitem__
backend = load_backend(db['ENGINE'])
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/db/utils.py", line 111, in load_backend
return import_module('%s.base' % backend_name)
File "/opt/python/pypy3.6-7.3.1/lib-python/3/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1003, in _gcd_import
File "<frozen importlib._bootstrap>", line 980, in _find_and_load
File "<frozen importlib._bootstrap>", line 964, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 674, in _load_unlocked
File "<builtin>/frozen importlib._bootstrap_external", line 691, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/travis/build/[secure]/CATMAID/django/lib/custom_postgresql_psycopg2/base.py", line 33, in <module>
from django.contrib.gis.db.backends.postgis.base import DatabaseWrapper as PostGISDatabaseWrapper
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
from .features import DatabaseFeatures
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/backends/base/features.py", line 3, in <module>
from django.contrib.gis.db.models import aggregates
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
import django.contrib.gis.db.models.functions # NOQA
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/functions.py", line 3, in <module>
from django.contrib.gis.db.models.fields import BaseSpatialField, GeometryField
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
from django.contrib.gis import forms, gdal
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
from .fields import ( # NOQA
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/forms/fields.py", line 2, in <module>
from django.contrib.gis.gdal import GDALException
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
from django.contrib.gis.gdal.datasource import DataSource
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
from django.contrib.gis.gdal.driver import Driver
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
File "/home/travis/virtualenv/pypy3.6-7.3.1/site-packages/django/contrib/gis/gdal/libgdal.py", line 46, in <module>
lgdal = CDLL(lib_path)
File "/opt/python/pypy3.6-7.3.1/lib-python/3/ctypes/__init__.py", line 350, in __init__
pypy_dll = _ffi.CDLL(name, mode)
OSError: Cannot load library libgdal.so.20: /usr/lib/libgdal.so.20: undefined symbol: sqlite3_column_table_name
PyPy 不与 libsqlite3 链接，但它包含基于 CFFI 的纯Python 模块lib_pypy/_sqlite3.py。尝试删除（或重命名）PyPy 自带的 libsqlite3.so 版本；这可能就足够了。如果不是，请继续阅读。
来自 lib_pypy/_sqlite3.py 的纯 Python 逻辑导入 _sqlite3_cffi.pypy-??.so，它由 CFFI 通过执行 _sqlite3_build.py 生成。 （CFFI 也包含在 PyPy 中。）所以以 root 身份重新 运行 pypy _sqlite3_build.py 应该使它重新生成 _sqlite3_cffi.*.so，使用系统提供的 libsqlite3.so 和 sqlite3.h.