使用 Bazel 为 AWS Lambda 创建 Python zip

Question

我有一个包含一组 Python AWS lambda 的 monorepo，我正在使用 Bazel for building and packaging the lambdas. I'm now trying to use Bazel to create a zip file that follows the expected AWS Lambdas packaging 并且我可以将其上传到 Lambda。想知道使用 Bazel 执行此操作的最佳方法是什么？

以下是我迄今为止尝试过的一些不同的东西：

尝试 1：py_binary

BUILD.bazel

py_binary(
name = "main_binary",
srcs = glob(["*.py"]),
main = "main.py",
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
        requirement("Faker"),
    ],
)

问题：

这会生成以下内容：

main_binary（python 可执行文件）
main_binary.runfiles
main_binary.runfiles_清单

Lambda 期望处理程序采用 lambda_function.lambda_handler 的格式。由于 main_binary 是可执行文件而不是 python 文件，它不会公开实际的处理程序方法，并且 lambda 会因为找不到它而崩溃。我尝试更新处理程序配置以简单地指向 main_binary 但它爆炸了，因为它需要两个参数（即 lambda_function.lambda_handler）。

尝试 2：py_library + pkg_zip

BUILD.bazel

py_library(
name = "main",
srcs = glob(["*.py"]),
visibility = ["//appcode/api/transaction_details:__subpackages__"],
deps = [
        requirement("Faker"),
    ],
)

pkg_zip(
name = "main_zip",
srcs =["//appcode/api/transaction_details/src:main" ],
)

问题：

这会生成一个 zip 文件：

main.py
__init__.py

zip 文件现在包括 main.py 但 none 的运行时依赖项。因此 lambda 爆炸了，因为它找不到 Faker.

其他尝试：

我也尝试过使用 --build_python_zip 标志以及带有通用规则的 @bazel_tools//tools/zip:zipper，但它们都会导致与前两次尝试相似的结果。

Answer 1

我们使用 @bazel_tools//tools/zip:zipper 和自定义规则。我们还通过 bazel 拉 serverless 使用 rules_nodejs 和运行它，这导致包构建发生在运行ning sls deploy.[=24= 之前]

我们使用 rules_python 中的 pip_parse。我不确定下面的 _short_path 函数是否适用于 pip_install 或其他机制。

支持文件过滤，虽然有点笨拙。理想情况下，zip 生成将由单独的二进制文件（即 Python 脚本）处理，这将允许使用常规 expressions/globs/etc 进行过滤。 Bazel 不支持 Starlark 中的正则表达式，所以我们使用自己的东西。

我摘录了一段：

lambda.bzl

"""
Support for serverless deployments.
"""

def contains(pattern):
    return "contains:" + pattern

def startswith(pattern):
    return "startswith:" + pattern

def endswith(pattern):
    return "endswith:" + pattern

def _is_ignored(path, patterns):
    for p in patterns:
        if p.startswith("contains:"):
            if p[len("contains:"):] in path:
                return True
        elif p.startswith("startswith:"):
            if path.startswith(p[len("startswith:"):]):
                return True
        elif p.startswith("endswith:"):
            if path.endswith(p[len("endswith:"):]):
                return True
        else:
            fail("Invalid pattern: " + p)

    return False

def _short_path(file_):
    # Remove prefixes for external and generated files.
    # E.g.,
    #   ../py_deps_pypi__pydantic/pydantic/__init__.py -> pydantic/__init__.py
    short_path = file_.short_path
    if short_path.startswith("../"):
        second_slash = short_path.index("/", 3)
        short_path = short_path[second_slash + 1:]
    return short_path

def _py_lambda_zip_impl(ctx):
    deps = ctx.attr.target[DefaultInfo].default_runfiles.files

    f = ctx.outputs.output

    args = []
    for dep in deps.to_list():
        short_path = _short_path(dep)

        # Skip ignored patterns
        if _is_ignored(short_path, ctx.attr.ignore):
            continue

        args.append(short_path + "=" + dep.path)

    ctx.actions.run(
        outputs = [f],
        inputs = deps,
        executable = ctx.executable._zipper,
        arguments = ["cC", f.path] + args,
        progress_message = "Creating archive...",
        mnemonic = "archiver",
    )

    out = depset(direct = [f])
    return [
        DefaultInfo(
            files = out,
        ),
        OutputGroupInfo(
            all_files = out,
        ),
    ]

_py_lambda_zip = rule(
    implementation = _py_lambda_zip_impl,
    attrs = {
        "target": attr.label(),
        "ignore": attr.string_list(),
        "_zipper": attr.label(
            default = Label("@bazel_tools//tools/zip:zipper"),
            cfg = "host",
            executable = True,
        ),
        "output": attr.output(),
    },
    executable = False,
    test = False,
)

def py_lambda_zip(name, target, ignore, **kwargs):
    _py_lambda_zip(
        name = name,
        target = target,
        ignore = ignore,
        output = name + ".zip",
        **kwargs
    )

BUILD.bazel

load("@npm_serverless//serverless:index.bzl", "serverless")
load(":lambda.bzl", "contains", "endswith", "py_lambda_zip", "startswith")

py_binary(
    name = "my_lambda_app",
    ...
)

py_lambda_zip(
    name = "lambda_archive",
    ignore = [
        contains("/__pycache__/"),
        endswith(".pyc"),
        endswith(".pyo"),
        
        # Ignore boto since it's provided by Lambda.
        startswith("boto3/"),
        startswith("botocore/"),

        # With the move to hermetic toolchains, the zip gets a lib/ directory containing the
        # python runtime. We don't need that.
        startswith("lib/"),
    ],
    target = ":my_lambda_app",

    # Only allow building on linux, since we don't want to upload a lambda zip file
    # with e.g. macos compiled binaries.
    target_compatible_with = [
        "@platforms//os:linux",
    ],
)

# The sls command requires that serverless.yml be in its working directory, and that the yaml file
# NOT be a symlink. So this target builds a directory containing a copy of serverless.yml, and also 
# symlinks the generated lambda_archive.zip in the same directory.
#
# It also generates a chdir.js script that we instruct node to execute to change to the proper working directory.
genrule(
    name = "sls_files",
    srcs = [
        "lambda_archive.zip",
        "serverless.yml",
    ],
    outs = [
        "sls_files/lambda_archive.zip",
        "sls_files/serverless.yml",
        "sls_files/chdir.js",
    ],
    cmd = """
        mkdir -p $(@D)/sls_files
        cp $(location serverless.yml) $(@D)/sls_files/serverless.yml
        cp -P $(location lambda_archive.zip) $(@D)/sls_files/lambda_archive.zip

        echo "const fs = require('fs');" \
             "const path = require('path');" \
             "process.chdir(path.dirname(fs.realpathSync(__filename)));" > $(@D)/sls_files/chdir.js
    """,
)

# Usage:
#   bazel run //:sls -- deploy <more args>
serverless(
    name = "sls",
    args = ["""--node_options=--require=./$(location sls_files/chdir.js)"""],
    data = [
        "sls_files/chdir.js",
        "sls_files/serverless.yml",
        "sls_files/lambda_archive.zip",
    ],
)

serverless.yml

service: my-app

package:
  artifact: lambda_archive.zip

# ... other config ...

Answer 2

以下是我对之前生成 lambda zip 的答案所做的更改。感谢@jvolkman 的原始建议。

project/BUILD.bazel: 添加了从 project/requirements.txt[= 生成 requirements_lock.txt 的规则18=]

load("@rules_python//python:pip.bzl", "compile_pip_requirements")

compile_pip_requirements(
    name = "requirements",
    extra_args = ["--allow-unsafe"],
    requirements_in = "requirements.txt",
    requirements_txt = "requirements_lock.txt",
)

project/WORKSPACE.bazel：将 pip_install 替换为 pip_parse

workspace(name = "mdc-eligibility")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "rules_python",
    sha256 = "9fcf91dbcc31fde6d1edb15f117246d912c33c36f44cf681976bd886538deba6",
    strip_prefix = "rules_python-0.8.0",
    url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.8.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
    name = "python3_9",
    python_version = "3.9",
)

load("@rules_python//python:pip.bzl", "pip_parse")
load("@python3_9//:defs.bzl", "interpreter")
pip_parse(
   name = "mndc-eligibility-deps",
   requirements_lock = "//:requirements_lock.txt",
   python_interpreter_target = interpreter,
   quiet = False
)
load("@mndc-eligibility-deps//:requirements.bzl", "install_deps")
install_deps()

project/build_rules/lambda_packaging/lambda.bzl: 修改了@jvolkman 提供的自定义规则，将源代码包含在生成的邮政编码中。

def contains(pattern):
    return "contains:" + pattern

def startswith(pattern):
    return "startswith:" + pattern

def endswith(pattern):
    return "endswith:" + pattern

def _is_ignored(path, patterns):
    for p in patterns:
        if p.startswith("contains:"):
            if p[len("contains:"):] in path:
                return True
        elif p.startswith("startswith:"):
            if path.startswith(p[len("startswith:"):]):
                return True
        elif p.startswith("endswith:"):
            if path.endswith(p[len("endswith:"):]):
                return True
        else:
            fail("Invalid pattern: " + p)

    return False

def _short_path(file_):
    # Remove prefixes for external and generated files.
    # E.g.,
    #   ../py_deps_pypi__pydantic/pydantic/__init__.py -> pydantic/__init__.py
    short_path = file_.short_path
    if short_path.startswith("../"):
        second_slash = short_path.index("/", 3)
        short_path = short_path[second_slash + 1:]
    return short_path

# steven chambers

def _py_lambda_zip_impl(ctx):
    deps = ctx.attr.target[DefaultInfo].default_runfiles.files

    f = ctx.outputs.output

    args = []
    for dep in deps.to_list():
        short_path = _short_path(dep)

        # Skip ignored patterns
        if _is_ignored(short_path, ctx.attr.ignore):
            continue

        args.append(short_path + "=" + dep.path)

    # MODIFICATION: Added source files to the map of files to zip
    source_files = ctx.attr.target[DefaultInfo].files
    for source_file in source_files.to_list():
        args.append(source_file.basename+"="+source_file.path)

    ctx.actions.run(
        outputs = [f],
        inputs = deps,
        executable = ctx.executable._zipper,
        arguments = ["cC", f.path] + args,
        progress_message = "Creating archive...",
        mnemonic = "archiver",
    )

    out = depset(direct = [f])
    return [
        DefaultInfo(
            files = out,
        ),
        OutputGroupInfo(
            all_files = out,
        ),
    ]

_py_lambda_zip = rule(
    implementation = _py_lambda_zip_impl,
    attrs = {
        "target": attr.label(),
        "ignore": attr.string_list(),
        "_zipper": attr.label(
            default = Label("@bazel_tools//tools/zip:zipper"),
            cfg = "host",
            executable = True,
        ),
        "output": attr.output(),
    },
    executable = False,
    test = False,
)

def py_lambda_zip(name, target, ignore, **kwargs):
    _py_lambda_zip(
        name = name,
        target = target,
        ignore = ignore,
        output = name + ".zip",
        **kwargs
    )

project/appcode/api/transaction_details/src/BUILD.bazel：使用自定义 py_lambda_zip 规则压缩 py_library

load("@mndc-eligibility-deps//:requirements.bzl", "requirement")
load("@python3_9//:defs.bzl", "interpreter")
load("//build_rules/lambda_packaging:lambda.bzl", "contains", "endswith", "py_lambda_zip", "startswith")

py_library(
    name = "main",
    srcs = glob(["*.py"]),
    visibility = ["//appcode/api/transaction_details:__subpackages__"],
    deps = [
            requirement("Faker"),
        ],
)

py_lambda_zip(
    name = "lambda_archive",
    ignore = [
        contains("/__pycache__/"),
        endswith(".pyc"),
        endswith(".pyo"),

        # Ignore boto since it's provided by Lambda.
        startswith("boto3/"),
        startswith("botocore/"),

        # With the move to hermetic toolchains, the zip gets a lib/ directory containing the
        # python runtime. We don't need that.
        startswith("lib/"),
    ],
    target = ":main",
)

使用 Bazel 为 AWS Lambda 创建 Python zip

Creating Python zip for AWS Lambda using Bazel

python

aws-lambda

bazel

bazel-rules

bazel-python