Poetry 如何处理二进制依赖项? (尤其是 numpy)
How does Poetry work regarding binary dependencies? (esp. numpy)
到目前为止,我一直在使用 conda 作为虚拟环境和依赖管理。但是,将我的 environment.yml 文件从我的开发机器传输到生产服务器时,有些东西没有按预期工作。现在,我想研究替代方案。诗歌似乎很好,尤其是因为
poetry also maintains a lock file, and it has a benefit over pipenv because it keeps track of which packages are subdependencies. (https://realpython.com/effective-python-environment/#poetry)
这可能会大大提高稳定性。但是,我正在研究 science-heavy projects(矩阵、数据科学、机器学习),所以在实践中我需要 scipy 堆栈(例如 numpy、pandas、scitkit-learn)。
Python became too slow for some pure computational workloads so numpy and scipy were born. [...] They are written in C and just wrapped as a python library.
Compiling such libraries brings a set of challenges since they (more or less) have to be compiled on your machine for maximum performance and proper linking with libraries like glibc.
Conda was introduced as an all-in-one solution to manage python environments for the scientific community.
[...] Instead of using a fragile process of compiling libraries on your machine, libraries are precompiled and just downloaded when you request them. Unfortunately, the solution comes with a caveat - conda does not use PyPI, the most popular index of python packages.
(https://modelpredict.com/python-dependency-management-tools#fnref:conda-compiling-challenges)
据我所知,这甚至不符合 Conda 标准,因为它做了相当多的优化以充分利用我的 CPU/GPU/architecture for numpy。 (https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/#Myth-#6:-Now-that-pip-uses-wheels,-conda-is-no-longer-necessary)
https://numpy.org/install/本身建议用conda,也说可以通过pip安装(poetry用的是pypi)
For users who know, from personal preference or reading about the main differences between conda and pip below, they prefer a pip/PyPI-based solution, we recommend:
[...] Use Poetry as the most well-maintained tool that provides a dependency resolver and environment management capabilities in a similar fashion as conda does.
我想了解 poetry 设置的稳定性和 conda 设置的速度。
诗歌如何处理二元依赖关系?
是不是也和conda一样考虑我的硬件?
如果poetry在这方面达不到,我可以和conda结合吗?
numpy
为不同的 os、cpu 体系结构和 python 版本提供了多个 wheel
文件。 wheel
包是预编译的,所以目标系统不需要编译包。
poetry
能够根据您的系统选择os适合您的方向盘。
说到这里,我推荐使用poetry
,只要你只需要python个包,pypi也有。一旦您需要其他非 python 工具,请坚持使用 conda。 (免责声明:我是 poetry
的维护者之一)。
在 MAC Os BigSur (11.1)
Python 3.9.1.6 和 poetry 1.1.4 的 numpy 安装失败(pandas)一整天我都遇到了问题。
我了解到poetry使用pip,当pip <= 20时不使用numpy的wheel版本。2.x。 Python 3.9.1.6 的全新安装不会升级 pip !
我就是这样安装我需要的东西的:
首先检查Python挂诗的默认版本(不合适重新安装)
poetry env info | grep -i python
Python: 3.9.1
Implementation: CPython
Python: /usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9
安装您的项目和 运行 poetry init
,无需任何您寻求的依赖项(您将在之后使用 poetry add <deps>
juste...)
你应该得到类似这样的东西 pyproject.toml :
[tool.poetry]
name = "project1"
version = "0.1.0"
description = ""
authors = [""]
[tool.poetry.dependencies]
python = "^3.9"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
启动诗歌shell
➜ project1 poetry shell
Python 2.7 will no longer be supported in the next feature release of Poetry (1.2).
You should consider updating your Python version to a supported one.
Note that you will still be able to manage Python 2.7 projects by using the env command.
See https://python-poetry.org/docs/managing-environments/ for more information.
Spawning shell within /Users/vincent/Library/Caches/pypoetry/virtualenvs/project1-5XOg8Qie-py3.9
➜ project1 . /Users/vincent/Library/Caches/pypoetry/virtualenvs/project1-5XOg8Qie-py3.9/bin/activate
(project1-5XOg8Qie-py3.9) ➜ project1
此时升级pip、wheel和setuptools!
(project1-5XOg8Qie-py3.9) ➜ project1 $ pip install --upgrade pip wheel setuptools
Looking in indexes: https://pypi.python.org/simple/
Collecting pip
Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
Collecting wheel
Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting setuptools
Using cached setuptools-51.1.2-py3-none-any.whl (784 kB)
Installing collected packages: pip, wheel, setuptools
Attempting uninstall: pip
Found existing installation: pip 20.2.4
Uninstalling pip-20.2.4:
Successfully uninstalled pip-20.2.4
Attempting uninstall: wheel
Found existing installation: wheel 0.35.1
Uninstalling wheel-0.35.1:
Successfully uninstalled wheel-0.35.1
Attempting uninstall: setuptools
Found existing installation: setuptools 50.3.2
Uninstalling setuptools-50.3.2:
Successfully uninstalled setuptools-50.3.2
Successfully installed pip-20.3.3 setuptools-51.1.2 wheel-0.36.2
现在安装应该可以了
(project1-5XOg8Qie-py3.9) ➜ project1 poetry install
Installing dependencies from lock file
Package operations: 5 installs, 0 updates, 0 removals
- Installing six (1.15.0)
- Installing numpy (1.19.5)
- Installing python-dateutil (2.8.1)
- Installing pytz (2020.5)
- Installing pandas (1.2.0)
Installing the current project: project1 (0.1.0)
哎呀!
到目前为止,我一直在使用 conda 作为虚拟环境和依赖管理。但是,将我的 environment.yml 文件从我的开发机器传输到生产服务器时,有些东西没有按预期工作。现在,我想研究替代方案。诗歌似乎很好,尤其是因为
poetry also maintains a lock file, and it has a benefit over pipenv because it keeps track of which packages are subdependencies. (https://realpython.com/effective-python-environment/#poetry)
这可能会大大提高稳定性。但是,我正在研究 science-heavy projects(矩阵、数据科学、机器学习),所以在实践中我需要 scipy 堆栈(例如 numpy、pandas、scitkit-learn)。
Python became too slow for some pure computational workloads so numpy and scipy were born. [...] They are written in C and just wrapped as a python library.
Compiling such libraries brings a set of challenges since they (more or less) have to be compiled on your machine for maximum performance and proper linking with libraries like glibc.
Conda was introduced as an all-in-one solution to manage python environments for the scientific community.
[...] Instead of using a fragile process of compiling libraries on your machine, libraries are precompiled and just downloaded when you request them. Unfortunately, the solution comes with a caveat - conda does not use PyPI, the most popular index of python packages.
(https://modelpredict.com/python-dependency-management-tools#fnref:conda-compiling-challenges)
据我所知,这甚至不符合 Conda 标准,因为它做了相当多的优化以充分利用我的 CPU/GPU/architecture for numpy。 (https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/#Myth-#6:-Now-that-pip-uses-wheels,-conda-is-no-longer-necessary)
https://numpy.org/install/本身建议用conda,也说可以通过pip安装(poetry用的是pypi)
For users who know, from personal preference or reading about the main differences between conda and pip below, they prefer a pip/PyPI-based solution, we recommend:
[...] Use Poetry as the most well-maintained tool that provides a dependency resolver and environment management capabilities in a similar fashion as conda does.
我想了解 poetry 设置的稳定性和 conda 设置的速度。
诗歌如何处理二元依赖关系? 是不是也和conda一样考虑我的硬件?
如果poetry在这方面达不到,我可以和conda结合吗?
numpy
为不同的 os、cpu 体系结构和 python 版本提供了多个 wheel
文件。 wheel
包是预编译的,所以目标系统不需要编译包。
poetry
能够根据您的系统选择os适合您的方向盘。
说到这里,我推荐使用poetry
,只要你只需要python个包,pypi也有。一旦您需要其他非 python 工具,请坚持使用 conda。 (免责声明:我是 poetry
的维护者之一)。
在 MAC Os BigSur (11.1)
Python 3.9.1.6 和 poetry 1.1.4 的 numpy 安装失败(pandas)一整天我都遇到了问题。
我了解到poetry使用pip,当pip <= 20时不使用numpy的wheel版本。2.x。 Python 3.9.1.6 的全新安装不会升级 pip !
我就是这样安装我需要的东西的:
首先检查Python挂诗的默认版本(不合适重新安装)
poetry env info | grep -i python
Python: 3.9.1
Implementation: CPython
Python: /usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9
安装您的项目和 运行 poetry init
,无需任何您寻求的依赖项(您将在之后使用 poetry add <deps>
juste...)
你应该得到类似这样的东西 pyproject.toml :
[tool.poetry]
name = "project1"
version = "0.1.0"
description = ""
authors = [""]
[tool.poetry.dependencies]
python = "^3.9"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
启动诗歌shell
➜ project1 poetry shell
Python 2.7 will no longer be supported in the next feature release of Poetry (1.2).
You should consider updating your Python version to a supported one.
Note that you will still be able to manage Python 2.7 projects by using the env command.
See https://python-poetry.org/docs/managing-environments/ for more information.
Spawning shell within /Users/vincent/Library/Caches/pypoetry/virtualenvs/project1-5XOg8Qie-py3.9
➜ project1 . /Users/vincent/Library/Caches/pypoetry/virtualenvs/project1-5XOg8Qie-py3.9/bin/activate
(project1-5XOg8Qie-py3.9) ➜ project1
此时升级pip、wheel和setuptools!
(project1-5XOg8Qie-py3.9) ➜ project1 $ pip install --upgrade pip wheel setuptools
Looking in indexes: https://pypi.python.org/simple/
Collecting pip
Using cached pip-20.3.3-py2.py3-none-any.whl (1.5 MB)
Collecting wheel
Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting setuptools
Using cached setuptools-51.1.2-py3-none-any.whl (784 kB)
Installing collected packages: pip, wheel, setuptools
Attempting uninstall: pip
Found existing installation: pip 20.2.4
Uninstalling pip-20.2.4:
Successfully uninstalled pip-20.2.4
Attempting uninstall: wheel
Found existing installation: wheel 0.35.1
Uninstalling wheel-0.35.1:
Successfully uninstalled wheel-0.35.1
Attempting uninstall: setuptools
Found existing installation: setuptools 50.3.2
Uninstalling setuptools-50.3.2:
Successfully uninstalled setuptools-50.3.2
Successfully installed pip-20.3.3 setuptools-51.1.2 wheel-0.36.2
现在安装应该可以了
(project1-5XOg8Qie-py3.9) ➜ project1 poetry install
Installing dependencies from lock file
Package operations: 5 installs, 0 updates, 0 removals
- Installing six (1.15.0)
- Installing numpy (1.19.5)
- Installing python-dateutil (2.8.1)
- Installing pytz (2020.5)
- Installing pandas (1.2.0)
Installing the current project: project1 (0.1.0)
哎呀!