使用无服务器和 python 3.7 在 AWS Lambda 中打包 dlib

Question

我正在尝试使用具有 python3.7 运行时的无服务器框架部署 AWS Lambda 函数，我需要将 dlib 打包为依赖项并导入到 lambda 函数中。让 dlib 在 python3.7 上工作并使用无服务器框架成功打包的最简单方法是什么？ TIA...

更新： 我已经将 dlib==19.9.0 添加到我的 requirements.txt，我正在使用无服务器插件 serverless-python-requirements 并且我使用基于 ubuntu 图像的 bitbucket 管道进行部署 python & 节点库。我还在管道脚本中安装了 cmake，因为 dlib 需要它来编译，正如我所见。

管道失败：

Container 'Build' exceeded memory limit.

并且 dlib 编译在 77% 处停止，并出现以下反复出现的弃用警告，直到管道超出内存并因失败而停止：

 Scanning dependencies of target dlib
.
.

 [ 77%] Building CXX object CMakeFiles/dlib_python.dir/src/other.cpp.o
    In file included from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/cast.h:16,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/tools/python/src/basic.cpp:3:
    /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/detail/internals.h:82:34: warning: ‘int PyThread_create_key()’ is deprecated [-Wdeprecated-declarations]
         decltype(PyThread_create_key()) tstate = 0; // Usually an int but a long on Cygwin64 with Python 3.x
                                      ^
    In file included from /usr/local/include/python3.7m/pystate.h:11,
                     from /usr/local/include/python3.7m/traceback.h:8,
                     from /usr/local/include/python3.7m/Python.h:119,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/detail/common.h:111,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pytypes.h:12,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/cast.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/tools/python/src/basic.cpp:3:
    /usr/local/include/python3.7m/pythread.h:95:17: note: declared here
     PyAPI_FUNC(int) PyThread_create_key(void) Py_DEPRECATED(3.7);
                     ^~~~~~~~~~~~~~~~~~~
    In file included from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/cast.h:16,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/attr.h:13,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/pybind11.h:43,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python/pybind_utils.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/dlib/../dlib/python.h:6,
                     from /tmp/pip-install-nx1hok9_/dlib/tools/python/src/basic.cpp:3:
    /tmp/pip-install-nx1hok9_/dlib/dlib/external/pybind11/include/pybind11/detail/internals.h:82:34: warning: ‘int PyThread_create_key()’ is deprecated [-Wdeprecated-declarations]
         decltype(PyThread_create_key()) tstate = 0; // Usually an int but a long on Cygwin64 with Python 3.x
                                      ^
    In file included from /usr/local/include/python3.7m/pystate.h:11,

在此处输入代码

Answer 1

Marcin 的建议可行，但有些乏味。幸运的是，无服务器框架来了。以下示例使用 Python 3.8，但可以轻松将其切换到 3.7。

先决条件：

如果你不是本地人，你需要 Docker 运行 linux
NPM
Serverless Python Requirements: npm install --save serverless-python-requirements

serverless.yml

service: dlib-example

provider:
  name: aws
  runtime: python3.8

functions:
  dlib:
    handler: handler.main
    layers:
      - {Ref: PythonRequirementsLambdaLayer}

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux
    layer: true

requirements.txt

dlib==19.19.0

handler.py

import dlib


def main(event, context):
    print(dlib.__version__)


if __name__ == "__main__":
    main('', '')

然后使用 sls deploy 构建依赖项（在 docker 容器中）并使用 CloudFormation 部署到 AWS。

测试

运行 sls invoke -f dlib --log 你会得到这样的东西：

null
--------------------------------------------------------------------
START RequestId: 9fba7253-2f3b-425f-a0b7-9ee3dfaec13b Version: $LATEST
19.19.0
END RequestId: 9fba7253-2f3b-425f-a0b7-9ee3dfaec13b
REPORT RequestId: 9fba7253-2f3b-425f-a0b7-9ee3dfaec13b  Duration: 1.66 ms   Billed Duration: 100 ms Memory Size: 1024 MB    Max Memory Used: 62 MB  Init Duration: 227.31 ms

干杯！

Answer 2

好的，我通过将管道大小增加到 2 倍解决了这个问题，它成功了

使用无服务器和 python 3.7 在 AWS Lambda 中打包 dlib

Package dlib in AWS Lambda using Serverless and python 3.7

amazon-web-services

aws-lambda

dlib

serverless-framework

python-3.7

serverless.yml

requirements.txt

handler.py

测试