如何在 CodePipeline 中处理多个环境?

How to handle multiple environments in CodePipeline?

我正在使用代码管道部署我的基础设施,我希望能够将它部署在不同的环境(开发、暂存、生产……)。

我目前有一个 buildspec.yml 文件,其中包含一些 "pip install" 指令和 "aws cloudformation package" 命令。我还创建了 2 个管道,一个用于生产,另一个用于开发,指向 github 上的 2 个不同分支。我遇到的问题是,由于在两个分支中文件都包含相似的资源,因此我在 S3 存储桶上发生了名称冲突。

使用 AWS CLI 和 cloudformation 创建或更新堆栈时,您可以使用 --parameters 选项传递参数。我想在我创建的 2 个管道中做类似的事情。

解决此问题的最佳解决方案是什么?

最终目标是自动化部署我们的基础设施。我们的基础架构由用户、KMS 密钥、Lamdbas(在 python 中)、组和存储桶组成。

我按照教程创建了两个管道:http://docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html

第一个管道链接到包含代码的 repo 的 master 分支,第二个管道链接到 staging 分支。我的目标是使用第一个管道在生产环境中自动部署主分支,并使用第二个管道在登台环境中自动部署登台分支。

我的 buildspec.yml 文件看起来像:

version: 0.1
phases:
    install:
        commands:
            - pip install requests -t .
            - pip install simplejson -t .
            - pip install Image -t .
            - aws cloudformation package --template-file image_processing_sam.yml --s3-bucket package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
    type: zip
    files:
        - new_image_processing_sam.yml

image_processing_sam.yml 文件看起来像:

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:

  ThumbnailFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: create_thumbnail.handler
      Runtime: python2.7
      Timeout: 30
      Description: "A function computing the thumbnail for an image."

  LambdaSecretEncryptionKey:
    Type: "AWS::KMS::Key"
    Properties:
      Description: "A key used to encrypt secrets used in the Lambda functions"
      Enabled: True
      EnableKeyRotation: False
      KeyPolicy:
        Version: "2012-10-17"
        Id: "lambda-secret-encryption-key"
        Statement:
          -
            Sid: "Allow administration of the key"
            Effect: "Allow"
            Principal:
              AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
            Action:
              - "kms:Create*"
              - "kms:Describe*"
              - "kms:Enable*"
              - "kms:List*"
              - "kms:Put*"
              - "kms:Update*"
              - "kms:Revoke*"
              - "kms:Disable*"
              - "kms:Get*"
              - "kms:Delete*"
              - "kms:ScheduleKeyDeletion"
              - "kms:CancelKeyDeletion"
            Resource: "*"
          -
            Sid: "Allow use of the key"
            Effect: "Allow"
            Principal:
              AWS:
                - !GetAtt LambdaExecutionRole.Arn
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
              - "kms:DescribeKey"
            Resource: "*"

  LambdaExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "LambdaExecutionRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - "lambda.amazonaws.com"
          Action:
          - "sts:AssumeRole"
      Policies:
        -
          PolicyName: LambdaKMS
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "kms:Decrypt"
                Resource: "*"
              -
                Effect: "Allow"
                Action:
                  - "lambda:InvokeFunction"
                Resource: "*"
      ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

  UserGroup:
      Type: "AWS::IAM::Group"

  LambdaTriggerUser:
    Type: "AWS::IAM::User"
    Properties:
      UserName: "LambdaTriggerUser"

  LambdaTriggerUserKeys:
    Type: "AWS::IAM::AccessKey"
    Properties:
      UserName:
        Ref: LambdaTriggerUser

  Users:
    Type: "AWS::IAM::UserToGroupAddition"
    Properties:
      GroupName:
        Ref: UserGroup
      Users:
        - Ref: LambdaTriggerUser

  Policies:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: UserPolicy
      PolicyDocument:
        Statement:
          -
            Effect: "Allow"
            Action:
              - "lambda:InvokeFunction"
            Resource:
              - !GetAtt DispatcherFunction.Arn
      Groups:
        - Ref: UserGroup

  PackageBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: "package-bucket"
      VersioningConfiguration:
        Status: "Enabled"

Outputs:
  LambdaTriggerUserAccessKey:
    Value:
      Ref: "LambdaTriggerUserKeys"
    Description: "AWSAccessKeyId of LambdaTriggerUser"

  LambdaTriggerUserSecretKey:
    Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
    Description: "AWSSecretKey of LambdaTriggerUser"

我在两个管道中都添加了部署操作,以执行在测试操作期间计算的更改集。

第一个管道工作得很好,可以完成我期望它做的所有事情。每次我在 master 分支中推送代码时,它都会被部署。

我面临的问题是,当我在暂存分支中推送代码时,管道中的所有内容都在运行,直到达到部署操作为止。部署操作尝试创建一个新堆栈,但由于它 buildspec.yml 和处理的 image_processing_sam.yml 完全相同,我遇到了如下名称冲突。

package-bucket already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaTriggerUser already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaExecutionRole already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
...

有没有办法参数化 buildspec.yml 以便能够为 image_processing_sam.yml 中的资源名称添加后缀?欢迎任何其他实现此目的的想法。

此致。

查看 Eric Nord 的回答。就是你要找的那个。


我也在 AWS 论坛上问过这个问题 here

AWS提供的解决方案如下:

Hi,

If your goal is to have different bucket names for staging and master, then another option is to use CloudFormation parameters.

When editing an existing pipeline if you edit an action you can expand the "Advanced" panel and enter parameter overrides to specify a different bucket prefix for each stage. You can also enter parameters as a separate .json file in your artifact.

There's more details on doing that here: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html

Here's a full walk through with a different stack configuration for test and production: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-basic-walkthrough.html

  • Tim.

您绝对应该按照提供的文档进行操作。这是我在下面提出的解决方案。


这是我自己的解决方案,我不满意。

我在构建时添加了一个脚本 运行 并根据构建项目的 CodeBuild 代理的 ARN 修改了我的模板。

我在可能发生命名冲突的地方添加了“BRANCH_NAME”。 image_processing_sam.yml 现在是:

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:

  ThumbnailFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: create_thumbnail.handler
      Runtime: python2.7
      Timeout: 30
      Description: "A function computing the thumbnail for an image."

  LambdaSecretEncryptionKey:
    Type: "AWS::KMS::Key"
    Properties:
      Description: "A key used to encrypt secrets used in the Lambda functions"
      Enabled: True
      EnableKeyRotation: False
      KeyPolicy:
        Version: "2012-10-17"
        Id: "lambda-secret-encryption-keyBRANCH_NAME"
        Statement:
          -
            Sid: "Allow administration of the key"
            Effect: "Allow"
            Principal:
              AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
            Action:
              - "kms:Create*"
              - "kms:Describe*"
              - "kms:Enable*"
              - "kms:List*"
              - "kms:Put*"
              - "kms:Update*"
              - "kms:Revoke*"
              - "kms:Disable*"
              - "kms:Get*"
              - "kms:Delete*"
              - "kms:ScheduleKeyDeletion"
              - "kms:CancelKeyDeletion"
            Resource: "*"
          -
            Sid: "Allow use of the key"
            Effect: "Allow"
            Principal:
              AWS:
                - !GetAtt LambdaExecutionRole.Arn
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
              - "kms:DescribeKey"
            Resource: "*"

  LambdaExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "LambdaExecutionRoleBRANCH_NAME"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - "lambda.amazonaws.com"
          Action:
          - "sts:AssumeRole"
      Policies:
        -
          PolicyName: LambdaKMSBRANCH_NAME
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "kms:Decrypt"
                Resource: "*"
              -
                Effect: "Allow"
                Action:
                  - "lambda:InvokeFunction"
                Resource: "*"
      ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

  UserGroup:
      Type: "AWS::IAM::Group"

  LambdaTriggerUser:
    Type: "AWS::IAM::User"
    Properties:
      UserName: "LambdaTriggerUserBRANCH_NAME"

  LambdaTriggerUserKeys:
    Type: "AWS::IAM::AccessKey"
    Properties:
      UserName:
        Ref: LambdaTriggerUser

  Users:
    Type: "AWS::IAM::UserToGroupAddition"
    Properties:
      GroupName:
        Ref: UserGroup
      Users:
        - Ref: LambdaTriggerUser

  Policies:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: UserPolicyBRANCH_NAME
      PolicyDocument:
        Statement:
          -
            Effect: "Allow"
            Action:
              - "lambda:InvokeFunction"
            Resource:
              - !GetAtt DispatcherFunction.Arn
      Groups:
        - Ref: UserGroup

  PackageBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: "package-bucketBRANCH_NAME"
      VersioningConfiguration:
        Status: "Enabled"

Outputs:
  LambdaTriggerUserAccessKey:
    Value:
      Ref: "LambdaTriggerUserKeys"
    Description: "AWSAccessKeyId of LambdaTriggerUser"

  LambdaTriggerUserSecretKey:
    Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
    Description: "AWSSecretKey of LambdaTriggerUser"

替换模板中“BRANCH_NAME”的script.sh是:

#!/bin/bash
echo $CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN
if [[ "$CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN" == *"master"* ]]; then
    sed "s/BRANCH_NAME//g" image_processing_sam.yml > generated_image_processing_sam.yml;
fi
if [[ "$CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN" == *"staging"* ]]; then
    sed "s/BRANCH_NAME/staging/g" image_processing_sam.yml > generated_image_processing_sam.yml;
fi

buildspec.yml现在是:

version: 0.1
phases:
    install:
        commands:
            # Install required module for python
            - pip install requests -t .
            - pip install simplejson -t .
            - pip install Image -t .
            - bash ./script.sh
            # To be able to see any issue in the generated template
            - cat generated_image_processing_sam.yml
            # Package the generated cloudformation template in order to deploy
            - aws cloudformation package --template-file generated_image_processing_sam.yml --s3-bucket piximate-package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
    type: zip
    files:
        - new_image_processing_sam.yml

希望对您有所帮助。如果有人可以提供任何可能有帮助的改进或文档,我将很高兴。

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-basic-walkthrough.html

模板配置文件通过如下参数文件应用于 CodePipeline 中的 CloudFormation:

{
  "Parameters" : {
    "DBName" : "TestWordPressDB",
    "DBPassword" : "TestDBRootPassword",
    "DBRootPassword" : "TestDBRootPassword",
    "DBUser" : "TestDBuser",    
    "KeyName" : "TestEC2KeyName"
    }
}

将这些文件放在你的 repo 的根目录下,它们至少可以通过两种方式被引用。

在您的 CodePipeline CloudFormation 中:

Configuration:
    ActionMode: REPLACE_ON_FAILURE
    RoleArn: !GetAtt [CFNRole, Arn]
    StackName: !Ref TestStackName
    TemplateConfiguration: !Sub "TemplateSource::${TestStackConfig}"
    TemplatePath: !Sub "TemplateSource::${TemplateFileName}"

或者在控制台的模板配置字段中:

值得注意的是,配置文件格式不同于通过 cli 使用的 CloudFormation

-- parameters

--参数使用这种格式:

[
  {
    "ParameterKey": "team",
    "ParameterValue": "AD-Student Life Applications"
  },
  {
    "ParameterKey": "env",
    "ParameterValue": "dev"
  },
  {
    "ParameterKey": "dataSensitivity",
    "ParameterValue": "public"
  },
  {
    "ParameterKey": "app",
    "ParameterValue": "events-list-test"
  }
]

CodePipeline Cloudformation 模板配置文件使用此格式:

{
  "Parameters" : {
    "DBName" : "TestWordPressDB",
    "DBPassword" : "TestDBRootPassword",
    "DBRootPassword" : "TestDBRootPassword",
    "DBUser" : "TestDBuser",    
    "KeyName" : "TestEC2KeyName"
  }
}