ModuleNotFoundError setup.py 使用已编译的 pyc 模块
ModuleNotFoundError with setup.py using a compiled pyc module
我通常可以导入已编译的 .pyc 模块以及 .py，但是当尝试使用 setup.py 打包一个简单的项目时，我得到 ModuleNotFoundError 已编译 .pyc 模块的异常。因为只有在使用 setup.py 时才会发生这种情况，否则工作正常，我不知道是否需要 setup.py 才能使这项工作正常进行。
项目结构目前是这样的：
proj
├── FAILING.pyc
├── __init__.py
├── aux/
│ ├── __init__.py
│ └── aux.c
└── main.py
和 setup.py:
from setuptools import setup, Extension, find_packages
DISTNAME = 'proj'
INSTALL_REQUIRES = [
'cython>=0.29.13',
'numpy>=1.16.4'
]
PYTHON_REQUIRES = '>=3.6'
ENTRY_POINTS = {
'console_scripts': ['proj = proj.main:main']
}
def setup_extensions(metadata):
ext_modules = [Extension('proj.aux.aux', sources=['proj/aux/aux.c'])]
metadata['ext_modules'] = ext_modules
def setup_package():
metadata = dict(
name=DISTNAME,
version='0.1',
package_dir={'': '.'},
packages=find_packages(),
entry_points=ENTRY_POINTS,
python_requires=PYTHON_REQUIRES,
install_requires=INSTALL_REQUIRES,
zip_safe=False,
)
setup_extensions(metadata)
setup(**metadata)
if __name__ == '__main__':
setup_package()
main.py:
#!/usr/bin/env python3
import proj.aux.aux as aux
import proj.FAILING
def main():
print('Hello World')
如果我只是尝试在 repl 上导入 FAILING.pyc，一切都会按预期进行：
>>> import FAILING
>>>
但是如果我先 运行 python3 setup.py intall 然后调用 proj 我会收到以下错误：
$ proj
Traceback (most recent call last):
File "/path/to/bin/proj", line 11, in <module>
load_entry_point('proj==0.1', 'console_scripts', 'proj')()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/path/to/lib/python3.8/site-packages/proj-0.1-py3.8-macosx-10.14-x86_64.egg/proj/main.py", line 4, in <module>
import proj.FAILING
ModuleNotFoundError: No module named 'proj.FAILING'
我也在 virtualenv 环境中 运行 宁此，虽然我猜这与错误无关。
我哪里做错了，或者我需要更改什么才能使这项工作正常进行？
这是一个小演示，它构建了一个包含 .pyc 文件的源代码分发和 wheel
请注意，我已经从您的示例中删除了大部分内容，因为 cython 内容与您的问题无关
set -euxo pipefail
rm -rf dist testpkg setup.py
cat > setup.py <<EOF
from setuptools import setup
setup(
name='foo',
version='1',
packages=['testpkg'],
package_data={'testpkg': ['*.pyc']},
)
EOF
mkdir testpkg
touch testpkg/__init__.py
echo 'print("hello hello world")' > testpkg/mod.py
python3 -m compileall -b testpkg/mod.py
rm testpkg/mod.py
python3 setup.py sdist bdist_wheel
tar --list -f dist/*.tar.gz
unzip -l dist/*.whl
关于 setup.py 有几点需要注意：
- 我在
packages 中包含了包含 .pyc 文件的包 -- 我本可以使用 setuptools.find_packages 代替，但这更简单
.pyc 文件包含为 package_data -- 默认情况下 pyc 文件不打包，因为它们通常是剩余的构建工件- 我需要使用
python3 -m compileall 的 -b 标志将 pyc 编译到遗留位置
- 即使是“已编译”的 pyc 文件也不会混淆实际代码，例如可以使用
dis 恢复它——当你在这里谈论“已编译”时，它只是意味着它已经被转换成（还是比较高级的）python字节码
您可以从源分发版或 wheel 安装包。
例如运行脚本：
$ bash t.sh
+ rm -rf dist testpkg setup.py
+ cat
+ mkdir testpkg
+ touch testpkg/__init__.py
+ echo 'print("hello hello world")'
+ python3 -m compileall -b testpkg/mod.py
Compiling 'testpkg/mod.py'...
+ rm testpkg/mod.py
+ python3 setup.py sdist bdist_wheel
running sdist
running egg_info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
running check
warning: check: missing required meta-data: url
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
creating foo-1
creating foo-1/foo.egg-info
creating foo-1/testpkg
copying files to foo-1...
copying setup.py -> foo-1
copying foo.egg-info/PKG-INFO -> foo-1/foo.egg-info
copying foo.egg-info/SOURCES.txt -> foo-1/foo.egg-info
copying foo.egg-info/dependency_links.txt -> foo-1/foo.egg-info
copying foo.egg-info/top_level.txt -> foo-1/foo.egg-info
copying testpkg/__init__.py -> foo-1/testpkg
copying testpkg/mod.pyc -> foo-1/testpkg
Writing foo-1/setup.cfg
creating dist
Creating tar archive
removing 'foo-1' (and everything under it)
running bdist_wheel
running build
running build_py
copying testpkg/__init__.py -> build/lib/testpkg
copying testpkg/mod.pyc -> build/lib/testpkg
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/__init__.py -> build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/mod.pyc -> build/bdist.linux-x86_64/wheel/testpkg
running install_egg_info
Copying foo.egg-info to build/bdist.linux-x86_64/wheel/foo-1-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/foo-1.dist-info/WHEEL
creating 'dist/foo-1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'testpkg/__init__.py'
adding 'testpkg/mod.pyc'
adding 'foo-1.dist-info/METADATA'
adding 'foo-1.dist-info/WHEEL'
adding 'foo-1.dist-info/top_level.txt'
adding 'foo-1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ tar --list -f dist/foo-1.tar.gz
foo-1/
foo-1/PKG-INFO
foo-1/foo.egg-info/
foo-1/foo.egg-info/PKG-INFO
foo-1/foo.egg-info/SOURCES.txt
foo-1/foo.egg-info/dependency_links.txt
foo-1/foo.egg-info/top_level.txt
foo-1/setup.cfg
foo-1/setup.py
foo-1/testpkg/
foo-1/testpkg/__init__.py
foo-1/testpkg/mod.pyc
+ unzip -l dist/foo-1-py3-none-any.whl
Archive: dist/foo-1-py3-none-any.whl
Length Date Time Name
--------- ---------- ----- ----
0 2021-02-17 22:27 testpkg/__init__.py
136 2021-02-17 22:27 testpkg/mod.pyc
163 2021-02-17 22:28 foo-1.dist-info/METADATA
92 2021-02-17 22:28 foo-1.dist-info/WHEEL
8 2021-02-17 22:27 foo-1.dist-info/top_level.txt
408 2021-02-17 22:28 foo-1.dist-info/RECORD
--------- -------
807 6 files
之后，我可以安装这个包并使用它：
$ mkdir t
$ cd t
$ virtualenv venv
...
$ . venv/bin/activate
$ pip install ../dist/foo-1-py3-none-any.whl
...
$ python3 -c 'import testpkg.mod'
hello hello world
我通常可以导入已编译的 .pyc 模块以及 .py，但是当尝试使用 setup.py 打包一个简单的项目时，我得到 ModuleNotFoundError 已编译 .pyc 模块的异常。因为只有在使用 setup.py 时才会发生这种情况，否则工作正常，我不知道是否需要 setup.py 才能使这项工作正常进行。
项目结构目前是这样的：
proj
├── FAILING.pyc
├── __init__.py
├── aux/
│ ├── __init__.py
│ └── aux.c
└── main.py
和 setup.py:
from setuptools import setup, Extension, find_packages
DISTNAME = 'proj'
INSTALL_REQUIRES = [
'cython>=0.29.13',
'numpy>=1.16.4'
]
PYTHON_REQUIRES = '>=3.6'
ENTRY_POINTS = {
'console_scripts': ['proj = proj.main:main']
}
def setup_extensions(metadata):
ext_modules = [Extension('proj.aux.aux', sources=['proj/aux/aux.c'])]
metadata['ext_modules'] = ext_modules
def setup_package():
metadata = dict(
name=DISTNAME,
version='0.1',
package_dir={'': '.'},
packages=find_packages(),
entry_points=ENTRY_POINTS,
python_requires=PYTHON_REQUIRES,
install_requires=INSTALL_REQUIRES,
zip_safe=False,
)
setup_extensions(metadata)
setup(**metadata)
if __name__ == '__main__':
setup_package()
main.py:
#!/usr/bin/env python3
import proj.aux.aux as aux
import proj.FAILING
def main():
print('Hello World')
如果我只是尝试在 repl 上导入 FAILING.pyc，一切都会按预期进行：
>>> import FAILING
>>>
但是如果我先 运行 python3 setup.py intall 然后调用 proj 我会收到以下错误：
$ proj
Traceback (most recent call last):
File "/path/to/bin/proj", line 11, in <module>
load_entry_point('proj==0.1', 'console_scripts', 'proj')()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/path/to/lib/python3.8/site-packages/proj-0.1-py3.8-macosx-10.14-x86_64.egg/proj/main.py", line 4, in <module>
import proj.FAILING
ModuleNotFoundError: No module named 'proj.FAILING'
我也在 virtualenv 环境中 运行 宁此，虽然我猜这与错误无关。
我哪里做错了，或者我需要更改什么才能使这项工作正常进行？
这是一个小演示，它构建了一个包含 .pyc 文件的源代码分发和 wheel
请注意，我已经从您的示例中删除了大部分内容，因为 cython 内容与您的问题无关
set -euxo pipefail
rm -rf dist testpkg setup.py
cat > setup.py <<EOF
from setuptools import setup
setup(
name='foo',
version='1',
packages=['testpkg'],
package_data={'testpkg': ['*.pyc']},
)
EOF
mkdir testpkg
touch testpkg/__init__.py
echo 'print("hello hello world")' > testpkg/mod.py
python3 -m compileall -b testpkg/mod.py
rm testpkg/mod.py
python3 setup.py sdist bdist_wheel
tar --list -f dist/*.tar.gz
unzip -l dist/*.whl
关于 setup.py 有几点需要注意：
- 我在
packages中包含了包含.pyc文件的包 -- 我本可以使用setuptools.find_packages代替，但这更简单 .pyc文件包含为package_data-- 默认情况下pyc文件不打包，因为它们通常是剩余的构建工件- 我需要使用
python3 -m compileall 的 - 即使是“已编译”的 pyc 文件也不会混淆实际代码，例如可以使用
dis恢复它——当你在这里谈论“已编译”时，它只是意味着它已经被转换成（还是比较高级的）python字节码
-b 标志将 pyc 编译到遗留位置
您可以从源分发版或 wheel 安装包。
例如运行脚本：
$ bash t.sh
+ rm -rf dist testpkg setup.py
+ cat
+ mkdir testpkg
+ touch testpkg/__init__.py
+ echo 'print("hello hello world")'
+ python3 -m compileall -b testpkg/mod.py
Compiling 'testpkg/mod.py'...
+ rm testpkg/mod.py
+ python3 setup.py sdist bdist_wheel
running sdist
running egg_info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
running check
warning: check: missing required meta-data: url
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
creating foo-1
creating foo-1/foo.egg-info
creating foo-1/testpkg
copying files to foo-1...
copying setup.py -> foo-1
copying foo.egg-info/PKG-INFO -> foo-1/foo.egg-info
copying foo.egg-info/SOURCES.txt -> foo-1/foo.egg-info
copying foo.egg-info/dependency_links.txt -> foo-1/foo.egg-info
copying foo.egg-info/top_level.txt -> foo-1/foo.egg-info
copying testpkg/__init__.py -> foo-1/testpkg
copying testpkg/mod.pyc -> foo-1/testpkg
Writing foo-1/setup.cfg
creating dist
Creating tar archive
removing 'foo-1' (and everything under it)
running bdist_wheel
running build
running build_py
copying testpkg/__init__.py -> build/lib/testpkg
copying testpkg/mod.pyc -> build/lib/testpkg
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/__init__.py -> build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/mod.pyc -> build/bdist.linux-x86_64/wheel/testpkg
running install_egg_info
Copying foo.egg-info to build/bdist.linux-x86_64/wheel/foo-1-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/foo-1.dist-info/WHEEL
creating 'dist/foo-1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'testpkg/__init__.py'
adding 'testpkg/mod.pyc'
adding 'foo-1.dist-info/METADATA'
adding 'foo-1.dist-info/WHEEL'
adding 'foo-1.dist-info/top_level.txt'
adding 'foo-1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ tar --list -f dist/foo-1.tar.gz
foo-1/
foo-1/PKG-INFO
foo-1/foo.egg-info/
foo-1/foo.egg-info/PKG-INFO
foo-1/foo.egg-info/SOURCES.txt
foo-1/foo.egg-info/dependency_links.txt
foo-1/foo.egg-info/top_level.txt
foo-1/setup.cfg
foo-1/setup.py
foo-1/testpkg/
foo-1/testpkg/__init__.py
foo-1/testpkg/mod.pyc
+ unzip -l dist/foo-1-py3-none-any.whl
Archive: dist/foo-1-py3-none-any.whl
Length Date Time Name
--------- ---------- ----- ----
0 2021-02-17 22:27 testpkg/__init__.py
136 2021-02-17 22:27 testpkg/mod.pyc
163 2021-02-17 22:28 foo-1.dist-info/METADATA
92 2021-02-17 22:28 foo-1.dist-info/WHEEL
8 2021-02-17 22:27 foo-1.dist-info/top_level.txt
408 2021-02-17 22:28 foo-1.dist-info/RECORD
--------- -------
807 6 files
之后，我可以安装这个包并使用它：
$ mkdir t
$ cd t
$ virtualenv venv
...
$ . venv/bin/activate
$ pip install ../dist/foo-1-py3-none-any.whl
...
$ python3 -c 'import testpkg.mod'
hello hello world