通过 Cloudformation、CodeBuild 和 CodePipeline 将 python 包部署到 AWS Lambda
Deploying a python package to AWS Lambda via Cloudformation, CodeBuild and CodePipeline
我想为我的 AWS 基础设施和 AWS Lambda 函数设置一个 CI/CD 管道。这个想法是让一切都在代码中,版本控制和自动化。我只想 git push
到存储库并让 CodePipeline 从那里接管,更新我的基础设施,运行 测试,如果成功,用最新代码更新我的 Lambda 函数。
我的 CloudFormation 模板基于 this excellent example。它看起来像这样:
AWSTemplateFormatVersion: 2010-09-09
Description: playground pipeline 1
Parameters:
SourceRepositoryName:
Type: String
Default: lambda-playground
SourceBranchName:
Type: String
Default: master
Resources:
ArtifactsBucket:
Type: AWS::S3::Bucket
DependsOn: CloudFormationRole
DeletionPolicy: Delete
Properties:
BucketName: lambda-playground-artifacts
CodeBuildRole:
Type: AWS::IAM::Role
DependsOn: CloudFormationRole
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- codebuild.amazonaws.com
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: CloudWatchWriteLogsPolicy
Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: '*'
- Sid: CodeCommitPullPolicy
Effect: Allow
Action:
- codecommit:GitPull
Resource: '*'
- Sid: S3GetObjectPolicy
Effect: Allow
Action:
- s3:GetObject
- s3:GetObjectVersion
Resource: '*'
- Sid: S3PutObjectPolicy
Effect: Allow
Action:
- s3:PutObject
Resource: '*'
CodePipelineRole:
Type: AWS::IAM::Role
DependsOn: CloudFormationRole
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- codepipeline.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
CloudFormationRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- cloudformation.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
CodeCommitRepository:
Type: AWS::CodeCommit::Repository
Properties:
RepositoryName: !Ref SourceRepositoryName
CodeBuildProject:
Type: AWS::CodeBuild::Project
DependsOn: CloudFormationRole
Properties:
Description: A playground of Lambda
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/python:2.7.12
Type: LINUX_CONTAINER
Name: lambda-playground
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: CODEPIPELINE
TimeoutInMinutes: 5
CodePipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
ArtifactStore:
Type: S3
Location: !Ref ArtifactsBucket
Name: !Ref AWS::StackName
RestartExecutionOnUpdate: true
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- Name: Source
ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeCommit
Version: 1
Configuration:
RepositoryName: !Ref SourceRepositoryName
BranchName: !Ref SourceBranchName
OutputArtifacts:
- Name: SourceOutput
- Name: PipelineDeploy
Actions:
- Name: UpdatePipeline
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Ref AWS::StackName
TemplatePath: SourceOutput::infra.yml
InputArtifacts:
- Name: SourceOutput
- Name: Build
Actions:
- Name: BuildAndTest
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref CodeBuildProject
InputArtifacts:
- Name: SourceOutput
OutputArtifacts:
- Name: BuildOutput
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref BuildOutput # DOES NOT WORK
FunctionName: playground-fc
Handler: src.main.handler
# TODO: Role: foo
Runtime: python2.7
Outputs:
ArtifactsBucketURL:
Description: Artifacts bucket URL
Value: !GetAtt ArtifactsBucket.WebsiteURL
RepositoryURL:
Description: SSH URL of the repository
Value: !GetAtt CodeCommitRepository.CloneUrlSsh
所以我有一个包含 3 个阶段的 CodePipeline - Source
,它从 CodeCommit 存储库中获取代码,PipelineDeploy
,它在必要时更新我的 CloudFormation 堆栈,Build
,它运行配置的 CodeBuild 项目。
我的 buildspec.yml 在这里:
version: 0.1
phases:
install:
commands:
- pip install -r requirements.txt -t lib
pre_build:
commands:
- python lib/pytest.py src
artifacts:
type: zip
files:
- src/**/*
- lib/**/*
它只是安装必要的库,通过 pytest 运行测试并创建部署 zip。此 zip 文件是 Build
阶段的 OutputArtifact
,并存储在 ArtifactsBucket
中。但是,每次它都会获得一个唯一的名称(例如 dfVV6Uh
),这是有道理的,但我不知道如何在 LambdaFunction -> Properties -> Code -> S3Key 字段中引用它。
所以我的问题是,如何创建 stack/pipeline,在完成所有步骤后,将最新版本部署到我的 AWS Lambda 函数?有没有办法可以使用 CodeDeploy 来做到这一点?这里的最佳做法是什么?
有一个示例说明如何实现类似的功能(通过 CodePipeline/CodeBuild 部署 lambda 函数)。
http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html
这个例子是用NodeJS写的lambda函数,但是基本思路是一样的。在通过 CodeBuild 构建工件并让 CodePipeline 在阶段内管理工件传播后,您可以使用 CloudFormation deploy/update lambda 函数。
如果有帮助请告诉我。
您可以使用 Parameter Override
with Fn::GetArtifactAtt
和 ObjectKey
属性将 AWS CodePipeline 生成的工件 .zip
的名称动态提供给您的 CloudFormation 部署操作。
使用您的示例,UpdatePipeline
CloudFormation 部署操作的配置如下所示:
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Ref AWS::StackName
TemplatePath: SourceOutput::infra.yml
ParameterOverrides:
{
"LambdaKey" : { "Fn::GetArtifactAtt" : ["LambdaFunctionSource", "ObjectKey"]}
}
InputArtifacts:
- Name: SourceOutput
- Name: BuildOutput
然后,在您的 CloudFormation 堆栈模板中声明并引用 LambdaKey
参数:
Parameters:
LambdaKey:
Type: String
# ...
Resources:
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref LambdaKey
# ...
我想为我的 AWS 基础设施和 AWS Lambda 函数设置一个 CI/CD 管道。这个想法是让一切都在代码中,版本控制和自动化。我只想 git push
到存储库并让 CodePipeline 从那里接管,更新我的基础设施,运行 测试,如果成功,用最新代码更新我的 Lambda 函数。
我的 CloudFormation 模板基于 this excellent example。它看起来像这样:
AWSTemplateFormatVersion: 2010-09-09
Description: playground pipeline 1
Parameters:
SourceRepositoryName:
Type: String
Default: lambda-playground
SourceBranchName:
Type: String
Default: master
Resources:
ArtifactsBucket:
Type: AWS::S3::Bucket
DependsOn: CloudFormationRole
DeletionPolicy: Delete
Properties:
BucketName: lambda-playground-artifacts
CodeBuildRole:
Type: AWS::IAM::Role
DependsOn: CloudFormationRole
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- codebuild.amazonaws.com
Policies:
- PolicyName: ServiceRole
PolicyDocument:
Version: 2012-10-17
Statement:
- Sid: CloudWatchWriteLogsPolicy
Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: '*'
- Sid: CodeCommitPullPolicy
Effect: Allow
Action:
- codecommit:GitPull
Resource: '*'
- Sid: S3GetObjectPolicy
Effect: Allow
Action:
- s3:GetObject
- s3:GetObjectVersion
Resource: '*'
- Sid: S3PutObjectPolicy
Effect: Allow
Action:
- s3:PutObject
Resource: '*'
CodePipelineRole:
Type: AWS::IAM::Role
DependsOn: CloudFormationRole
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- codepipeline.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
CloudFormationRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- sts:AssumeRole
Principal:
Service:
- cloudformation.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AdministratorAccess
CodeCommitRepository:
Type: AWS::CodeCommit::Repository
Properties:
RepositoryName: !Ref SourceRepositoryName
CodeBuildProject:
Type: AWS::CodeBuild::Project
DependsOn: CloudFormationRole
Properties:
Description: A playground of Lambda
Artifacts:
Type: CODEPIPELINE
Environment:
ComputeType: BUILD_GENERAL1_SMALL
Image: aws/codebuild/python:2.7.12
Type: LINUX_CONTAINER
Name: lambda-playground
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: CODEPIPELINE
TimeoutInMinutes: 5
CodePipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
ArtifactStore:
Type: S3
Location: !Ref ArtifactsBucket
Name: !Ref AWS::StackName
RestartExecutionOnUpdate: true
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Name: Source
Actions:
- Name: Source
ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeCommit
Version: 1
Configuration:
RepositoryName: !Ref SourceRepositoryName
BranchName: !Ref SourceBranchName
OutputArtifacts:
- Name: SourceOutput
- Name: PipelineDeploy
Actions:
- Name: UpdatePipeline
ActionTypeId:
Category: Deploy
Owner: AWS
Provider: CloudFormation
Version: 1
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Ref AWS::StackName
TemplatePath: SourceOutput::infra.yml
InputArtifacts:
- Name: SourceOutput
- Name: Build
Actions:
- Name: BuildAndTest
ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref CodeBuildProject
InputArtifacts:
- Name: SourceOutput
OutputArtifacts:
- Name: BuildOutput
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref BuildOutput # DOES NOT WORK
FunctionName: playground-fc
Handler: src.main.handler
# TODO: Role: foo
Runtime: python2.7
Outputs:
ArtifactsBucketURL:
Description: Artifacts bucket URL
Value: !GetAtt ArtifactsBucket.WebsiteURL
RepositoryURL:
Description: SSH URL of the repository
Value: !GetAtt CodeCommitRepository.CloneUrlSsh
所以我有一个包含 3 个阶段的 CodePipeline - Source
,它从 CodeCommit 存储库中获取代码,PipelineDeploy
,它在必要时更新我的 CloudFormation 堆栈,Build
,它运行配置的 CodeBuild 项目。
我的 buildspec.yml 在这里:
version: 0.1
phases:
install:
commands:
- pip install -r requirements.txt -t lib
pre_build:
commands:
- python lib/pytest.py src
artifacts:
type: zip
files:
- src/**/*
- lib/**/*
它只是安装必要的库,通过 pytest 运行测试并创建部署 zip。此 zip 文件是 Build
阶段的 OutputArtifact
,并存储在 ArtifactsBucket
中。但是,每次它都会获得一个唯一的名称(例如 dfVV6Uh
),这是有道理的,但我不知道如何在 LambdaFunction -> Properties -> Code -> S3Key 字段中引用它。
所以我的问题是,如何创建 stack/pipeline,在完成所有步骤后,将最新版本部署到我的 AWS Lambda 函数?有没有办法可以使用 CodeDeploy 来做到这一点?这里的最佳做法是什么?
有一个示例说明如何实现类似的功能(通过 CodePipeline/CodeBuild 部署 lambda 函数)。 http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html
这个例子是用NodeJS写的lambda函数,但是基本思路是一样的。在通过 CodeBuild 构建工件并让 CodePipeline 在阶段内管理工件传播后,您可以使用 CloudFormation deploy/update lambda 函数。
如果有帮助请告诉我。
您可以使用 Parameter Override
with Fn::GetArtifactAtt
和 ObjectKey
属性将 AWS CodePipeline 生成的工件 .zip
的名称动态提供给您的 CloudFormation 部署操作。
使用您的示例,UpdatePipeline
CloudFormation 部署操作的配置如下所示:
Configuration:
ActionMode: CREATE_UPDATE
Capabilities: CAPABILITY_IAM
RoleArn: !GetAtt CloudFormationRole.Arn
StackName: !Ref AWS::StackName
TemplatePath: SourceOutput::infra.yml
ParameterOverrides:
{
"LambdaKey" : { "Fn::GetArtifactAtt" : ["LambdaFunctionSource", "ObjectKey"]}
}
InputArtifacts:
- Name: SourceOutput
- Name: BuildOutput
然后,在您的 CloudFormation 堆栈模板中声明并引用 LambdaKey
参数:
Parameters:
LambdaKey:
Type: String
# ...
Resources:
LambdaFunction:
Type: AWS::Lambda::Function
Properties:
Code:
S3Bucket: !Ref ArtifactsBucket
S3Key: !Ref LambdaKey
# ...