Serverless python 打包numpy依赖报错

Question

从我部署的 Python3.7 Lambda 函数调用函数时，我运行遇到了问题，根据错误消息，该函数与 numpy 相关。该问题指出无法导入包，尽管尝试了我已阅读的许多解决方案，但我没有取得任何成功，我想知道接下来要测试什么或如何进一步调试。

我试过以下方法：

安装Docker，添加插件serverless-python-requirements，在yml中配置
在应用程序目录中安装包以进行捆绑和部署，pip install -t src/vendor -r requirements.txt --no-cache-dir
已卸载 setuptools 和 numpy 并按顺序重新安装

错误消息（在运行 sls invoke -f auth 之后显示）：


{
    "errorMessage": "Unable to import module 'data': Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy c-extensions failed.\n- Try uninstalling and reinstalling numpy.\n- If you have already done that, then:\n  1. Check that you expected to use Python3.7 from \"/var/lang/bin/python3.7\",\n     and that you have no directories in your PATH or PYTHONPATH that can\n     interfere with the Python and numpy version \"1.18.1\" you're trying to use.\n  2. If (1) looks fine, you can open a new issue at\n     https://github.com/numpy/numpy/issues.  Please include details on:\n     - how you installed Python\n     - how you installed numpy\n     - your operating system\n     - whether or not you have multiple versions of Python installed\n     - if you built from source, your compiler versions and ideally a build log\n\n- If you're working with a numpy git repository, try `git clean -xdf`\n  (removes all files not under version control) and rebuild numpy.\n\nNote: this error has many possible causes, so please don't comment on\nan existing issue about this - open a new one instead.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
    "errorType": "Runtime.ImportModuleError"
}

提供的是我的设置：

OS: Mac OS X
Local Python: /Users/me/miniconda3/bin/python
Local Python version: Python 3.7.4

无服务器环境信息（运行时=Python3.7）：

 Operating System:          darwin
 Node Version:              12.14.0
 Framework Version:         1.67.3
 Plugin Version:            3.6.6
 SDK Version:               2.3.0
 Components Version:        2.29.1

Docker:

Docker version 19.03.13, build 4484c46d9d

serverless.yml:

service: understand-your-sleep-api

plugins:
  - serverless-python-requirements
  - serverless-offline-python

custom:
  pythonRequirements:
    dockerizePip: true # non-linux
    slim: true
    useStaticCache: false
    useDownloadCache: false
    invalidateCaches: true

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage, 'dev'}
  region: us-east-1
  iamRoleStatements:
    - Effect: Allow
      Action:
        - ssm:GetParameter
      Resource: "arn:aws:ssm:us-east-1:*id*:parameter/*"
  environment:
    STAGE: ${self:provider.stage}

functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true

package:
  exclude:
    - env.yml
    - node_modules/**

requirements.txt:

pandas==1.0.0
fitbit==0.3.1
oauthlib==3.1.0
requests==2.22.0
requests-oauthlib==1.3.0

data.py:

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow
import json
from datetime import timedelta, datetime, date
import math
import pandas as pd
from requests_oauthlib import OAuth2Session
from urllib.parse import urlparse, parse_qs
import fitbit
import requests
import webbrowser
import base64
import os 
import logging

def auth(event, context):
    ...

Answer 1

由于您使用的是 serverless-python-requirements 插件，它会为您打包库。换句话说，您不需要手动完成 pip install -t src/vendor -r requirements.txt --no-cache-dir 所有这些事情。

为了解决您的问题，删除 src/vendor 和 data.py 中的以下两行：

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow

然后坐下来，让 serverless-python-requirements 为您完成工作。

Answer 2

使用 lambda 层来打包你所有的需求，确保你在 requirements.txt 文件中有 numpy。尝试一次。
这仅在 serverless-python-requirements 插件列在插件部分时有效。
用这个替换您的自定义键并为函数提供使用该层的参考

custom:
  pythonRequirements:
    layer: true
functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true
    layers:
      - { Ref: PythonRequirementsLambdaLayer}

Answer 3

我检查了 zipinfo .requirements.zip，发现加载的是 macos dynlib 而不是 linux 所以文件

我使用 dockerizePip 解决了这个问题：非linux

请注意，如果在工作目录中 a .requirements.zip 已经存在，则不会触发此操作，因此 git clean -xfd 在运行 sls 部署之前

Serverless python 打包numpy依赖报错

Serverless python packaging numpy dependency error

numpy

serverless-framework

aws-serverless