无法为 ipopt、pyomo、cython 创建 AWS Lambda 部署包
Unable to create AWS Lambda Deployment Package for ipopt, pyomo, cython
我正在尝试创建一个 AWS Lambda 部署包，因此我可以将它用作 AWS Lambda 层。具体来说，我正在尝试将 ipopt sovler 与 pyomo 和 cython 结合使用。我可以毫无问题地打包 pyomo，但是当我尝试为包安装 pip install ipopt 时，出现错误：
ERROR: Command errored out with exit status 1:
command: 'c:\python36\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"'; __file__='"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\username\AppData\Local\Temp\pip-record-d1o_oqjy\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python36\Include\ipopt'
cwd: C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\
Complete output (20 lines):
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.6
creating build\lib.win-amd64-3.6\ipopt
copying ipopt\ipopt_wrapper.py -> build\lib.win-amd64-3.6\ipopt
copying ipopt\version.py -> build\lib.win-amd64-3.6\ipopt
copying ipopt\__init__.py -> build\lib.win-amd64-3.6\ipopt
running build_ext
skipping 'src\cyipopt.c' Cython extension (up-to-date)
building 'ipopt.cyipopt' extension
creating build\temp.win-amd64-3.6
creating build\temp.win-amd64-3.6\Release
creating build\temp.win-amd64-3.6\Release\src
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iinclude_mt/coin -Ic:\python36\lib\site-packages\numpy\core\include -Ic:\python36\include -Ic:\python36\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\include.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits.1\include\shared" "-IC:\Program Files (x86)\Windows Kits.1\include\um" "-IC:\Program Files (x86)\Windows Kits.1\include\winrt" /Tcsrc\cyipopt.c /Fobuild\temp.win-amd64-3.6\Release\src\cyipopt.obj
cyipopt.c
c:\python36\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
src\cyipopt.c(569): fatal error C1083: Cannot open include file: 'IpStdCInterface.h': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\python36\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"'; __file__='"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\username\AppData\Local\Temp\pip-record-d1o_oqjy\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python36\Include\ipopt' Check the logs for full command output.
我正在尝试 运行 AWS lambda 函数中的 pyomo/ipopt 包。这个包不在 SDK 中，所以我读到我必须将依赖项编译成一个 zip，然后将 zip 上传到 lambda 函数中。
我做了一些研究并尝试了下面的链接，但是当我尝试 pip install ipopt 时收到错误消息 - 我在 Windows 框上，但我也试过了在 Linux EC2 实例上
感谢您的帮助！
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/
https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e
https://github.com/coin-or/Ipopt
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
TL;DR，这是与 Python 3.6 运行时兼容的工作 lambda 部署包。 [gdrive download link]
事实证明，将所有需要的东西打包到一个 zip 中非常棘手。我花了很长时间（几个小时）手动将库文件复制过来。以下是我生成此包所采取的步骤。
- 启动本地 lambda Python 3.6 运行时 docker 容器
docker run -it --rm lambci/lambda:build-python3.6 bash
- pip 将所有依赖项安装到本地目录
mkdir ipopt-lib
PYTHONPATH=$PYTHONPATH:~/ipopt-lib
pip install Cython numpy -t ipopt-lib/
- 按照 official guide 从源代码构建和安装 Ipopt
yum install lapack-devel
git clone https://github.com/coin-or/Ipopt.git
cd Ipopt
IPOPTDIR=`pwd`
mkdir $IPOPTDIR/build
cd $IPOPTDIR/build
$IPOPTDIR/configure
make
make install
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pip install ipopt -t ipopt-lib/
手动复制所有库文件过来
cp -t ~/ipopt-lib/ /usr/local/lib/lib*
cp -t ~/ipopt-lib/ /usr/lib64/libblas.so* /usr/lib64/liblapack*
cp -t ~/ipopt-lib/ /usr/lib64/libgfortran.so.3*
cp -t ~/ipopt-lib/ /usr/lib64/libstdc++.so*
cp -t ~/ipopt-lib/ /usr/lib64/libquadmath.so.0*
- 一起压缩
cd ~/ipopt-lib
zip -r /root/ipopt-lib.zip *
- 复制到主机
docker cp <CONTAINER_ID>:/root/ipopt-lib.zip ~/Downloads/
以下是此部署包中可用的所有 Python 包（包括运行时安装的默认包）：
$ pip freeze
appdirs==1.4.3
arrow==0.15.5
attrs==19.3.0
aws-lambda-builders==0.8.0
aws-sam-cli==0.47.0
aws-sam-translator==1.22.0
awscli==1.18.41
binaryornot==0.4.4
boto3==1.12.41
botocore==1.15.41
certifi==2020.4.5.1
chardet==3.0.4
chevron==0.13.1
click==7.1.1
colorama==0.4.3
cookiecutter==1.6.0
Cython==0.29.17
dateparser==0.7.4
distlib==0.3.0
docker==4.2.0
docutils==0.15.2
filelock==3.0.12
Flask==1.0.4
future==0.18.2
idna==2.8
importlib-metadata==1.6.0
importlib-resources==1.4.0
ipopt==0.1.9
itsdangerous==1.1.0
Jinja2==2.11.2
jinja2-time==0.2.0
jmespath==0.9.5
jsonschema==3.2.0
MarkupSafe==1.1.1
numpy==1.18.3
pipenv==2018.11.26
poyo==0.5.0
pyasn1==0.4.8
pyrsistent==0.16.0
python-dateutil==2.8.0
pytz==2019.3
PyYAML==5.3.1
regex==2020.4.4
requests==2.22.0
rsa==3.4.2
s3transfer==0.3.3
serverlessrepo==0.1.9
six==1.14.0
tomlkit==0.5.8
tzlocal==2.0.0
urllib3==1.25.8
virtualenv==20.0.18
virtualenv-clone==0.5.4
websocket-client==0.57.0
Werkzeug==1.0.1
whichcraft==0.6.1
zipp==3.1.0
使用 Python 3.6 运行时
在 lambda 上进行测试
import sys
try:
sys.path.index('/opt')
except ValueError:
sys.path.append('/opt')
import os
os.environ['LD_LIBRARY_PATH'] += os.pathsep + '/opt'
import json
import numpy
import ipopt
def lambda_handler(event, context):
print(numpy.__version__)
print(ipopt.__version__)
结果
Response:
null
Request ID:
"9bd9fd88-d340-4976-a954-8a727e173933"
Function Logs:
START RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 Version: $LATEST
1.18.3
0.1.9
END RequestId: 9bd9fd88-d340-4976-a954-8a727e173933
REPORT RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 Duration: 0.45 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 74 MB Init Duration: 343.99 ms
干杯！
我正在尝试创建一个 AWS Lambda 部署包，因此我可以将它用作 AWS Lambda 层。具体来说，我正在尝试将 ipopt sovler 与 pyomo 和 cython 结合使用。我可以毫无问题地打包 pyomo，但是当我尝试为包安装 pip install ipopt 时，出现错误：
ERROR: Command errored out with exit status 1:
command: 'c:\python36\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"'; __file__='"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\username\AppData\Local\Temp\pip-record-d1o_oqjy\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python36\Include\ipopt'
cwd: C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\
Complete output (20 lines):
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.6
creating build\lib.win-amd64-3.6\ipopt
copying ipopt\ipopt_wrapper.py -> build\lib.win-amd64-3.6\ipopt
copying ipopt\version.py -> build\lib.win-amd64-3.6\ipopt
copying ipopt\__init__.py -> build\lib.win-amd64-3.6\ipopt
running build_ext
skipping 'src\cyipopt.c' Cython extension (up-to-date)
building 'ipopt.cyipopt' extension
creating build\temp.win-amd64-3.6
creating build\temp.win-amd64-3.6\Release
creating build\temp.win-amd64-3.6\Release\src
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iinclude_mt/coin -Ic:\python36\lib\site-packages\numpy\core\include -Ic:\python36\include -Ic:\python36\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\include.0.10240.0\ucrt" "-IC:\Program Files (x86)\Windows Kits.1\include\shared" "-IC:\Program Files (x86)\Windows Kits.1\include\um" "-IC:\Program Files (x86)\Windows Kits.1\include\winrt" /Tcsrc\cyipopt.c /Fobuild\temp.win-amd64-3.6\Release\src\cyipopt.obj
cyipopt.c
c:\python36\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
src\cyipopt.c(569): fatal error C1083: Cannot open include file: 'IpStdCInterface.h': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\python36\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"'; __file__='"'"'C:\Users\username\AppData\Local\Temp\pip-install-rir_unvz\ipopt\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\username\AppData\Local\Temp\pip-record-d1o_oqjy\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python36\Include\ipopt' Check the logs for full command output.
我正在尝试 运行 AWS lambda 函数中的 pyomo/ipopt 包。这个包不在 SDK 中，所以我读到我必须将依赖项编译成一个 zip，然后将 zip 上传到 lambda 函数中。
我做了一些研究并尝试了下面的链接，但是当我尝试 pip install ipopt 时收到错误消息 - 我在 Windows 框上，但我也试过了在 Linux EC2 实例上
感谢您的帮助！
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/
https://medium.com/@qtangs/creating-new-aws-lambda-layer-for-python-pandas-library-348b126e9f3e
https://github.com/coin-or/Ipopt
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
TL;DR，这是与 Python 3.6 运行时兼容的工作 lambda 部署包。 [gdrive download link]
事实证明，将所有需要的东西打包到一个 zip 中非常棘手。我花了很长时间（几个小时）手动将库文件复制过来。以下是我生成此包所采取的步骤。
- 启动本地 lambda Python 3.6 运行时 docker 容器
docker run -it --rm lambci/lambda:build-python3.6 bash
- pip 将所有依赖项安装到本地目录
mkdir ipopt-lib
PYTHONPATH=$PYTHONPATH:~/ipopt-lib
pip install Cython numpy -t ipopt-lib/
- 按照 official guide 从源代码构建和安装 Ipopt
yum install lapack-devel
git clone https://github.com/coin-or/Ipopt.git
cd Ipopt
IPOPTDIR=`pwd`
mkdir $IPOPTDIR/build
cd $IPOPTDIR/build
$IPOPTDIR/configure
make
make install
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig pip install ipopt -t ipopt-lib/手动复制所有库文件过来
cp -t ~/ipopt-lib/ /usr/local/lib/lib*
cp -t ~/ipopt-lib/ /usr/lib64/libblas.so* /usr/lib64/liblapack*
cp -t ~/ipopt-lib/ /usr/lib64/libgfortran.so.3*
cp -t ~/ipopt-lib/ /usr/lib64/libstdc++.so*
cp -t ~/ipopt-lib/ /usr/lib64/libquadmath.so.0*
- 一起压缩
cd ~/ipopt-lib
zip -r /root/ipopt-lib.zip *
- 复制到主机
docker cp <CONTAINER_ID>:/root/ipopt-lib.zip ~/Downloads/
以下是此部署包中可用的所有 Python 包（包括运行时安装的默认包）：
$ pip freeze
appdirs==1.4.3
arrow==0.15.5
attrs==19.3.0
aws-lambda-builders==0.8.0
aws-sam-cli==0.47.0
aws-sam-translator==1.22.0
awscli==1.18.41
binaryornot==0.4.4
boto3==1.12.41
botocore==1.15.41
certifi==2020.4.5.1
chardet==3.0.4
chevron==0.13.1
click==7.1.1
colorama==0.4.3
cookiecutter==1.6.0
Cython==0.29.17
dateparser==0.7.4
distlib==0.3.0
docker==4.2.0
docutils==0.15.2
filelock==3.0.12
Flask==1.0.4
future==0.18.2
idna==2.8
importlib-metadata==1.6.0
importlib-resources==1.4.0
ipopt==0.1.9
itsdangerous==1.1.0
Jinja2==2.11.2
jinja2-time==0.2.0
jmespath==0.9.5
jsonschema==3.2.0
MarkupSafe==1.1.1
numpy==1.18.3
pipenv==2018.11.26
poyo==0.5.0
pyasn1==0.4.8
pyrsistent==0.16.0
python-dateutil==2.8.0
pytz==2019.3
PyYAML==5.3.1
regex==2020.4.4
requests==2.22.0
rsa==3.4.2
s3transfer==0.3.3
serverlessrepo==0.1.9
six==1.14.0
tomlkit==0.5.8
tzlocal==2.0.0
urllib3==1.25.8
virtualenv==20.0.18
virtualenv-clone==0.5.4
websocket-client==0.57.0
Werkzeug==1.0.1
whichcraft==0.6.1
zipp==3.1.0
使用 Python 3.6 运行时
在 lambda 上进行测试import sys
try:
sys.path.index('/opt')
except ValueError:
sys.path.append('/opt')
import os
os.environ['LD_LIBRARY_PATH'] += os.pathsep + '/opt'
import json
import numpy
import ipopt
def lambda_handler(event, context):
print(numpy.__version__)
print(ipopt.__version__)
结果
Response:
null
Request ID:
"9bd9fd88-d340-4976-a954-8a727e173933"
Function Logs:
START RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 Version: $LATEST
1.18.3
0.1.9
END RequestId: 9bd9fd88-d340-4976-a954-8a727e173933
REPORT RequestId: 9bd9fd88-d340-4976-a954-8a727e173933 Duration: 0.45 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 74 MB Init Duration: 343.99 ms