允许VPC中的Lambda访问同一VPC中的Elasticsearch域

Question

我正在学习使用 Amazon 服务，特别是我目前想使用 Cloud Formation 脚本创建一个简单的设置：一个 VPC，带有一个用 JS 编写的单个 lambda，可以访问其中的 Elasticsearch 服务相同的 VPC。

不知怎么的，我无法让它工作。从 lambda 到 Elasticsearch 域的所有请求总是超时。但是，来自同一 VPC 中的 EC2 实例运行 Amazon Linux 2 的相同 JS 代码或 curl（即使没有任何额外授权，只是卷曲 ES 域端点）发出的相同请求工作正常我可以从那个 EC2 实例（通过 SSH 连接到它）与 Elasticsearch 进行良好的通信。

同时lambda可以访问VPC中的Aurora集群，所以lambda无法访问VPC资源不是一般问题

请告诉我我在 Cloud Formation 设置描述中做错了什么？下面是我的 Cloud Formation 模板的相关摘录和能够从 EC2 实例访问 ES 服务但不能对 lambda 执行相同操作的 JS 代码示例：

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join 
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'
      Path: /
      RoleName: !Join 
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join 
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        S3Bucket: !Ref SomeDeploymentBucket
        S3Key: >-
          tutorial/some-app/dev/1545610972669-2018-12-24T00:22:52.669Z/some-app.zip
      FunctionName: some-lambda
      Handler: app.server
      MemorySize: 1024
      Role: !GetAtt 
        - IamRoleLambdaExecution
        - Arn
      Runtime: nodejs8.10
      Timeout: 6
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20

    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'
      ElasticsearchVersion: 6.3
      ElasticsearchClusterConfig:
        InstanceCount: 2
        InstanceType: m3.medium.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet

JS代码（没有用creds签名的版本，但签名也不起作用）：

var es = require('elasticsearch');
var client = new es.Client({
    host: 'vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80',
    log: 'trace'
});

client.ping({
    requestTimeout: 1000
}, function(error, res, status){
    if(error) {
        console.trace('es cluster error!');
        console.trace(error);
    } else {
        console.log('All is well');
        var response = {
            error: error,
            res: res,
            status: status
        }
        console.log(JSON.stringify(response));
    }
});

从同一 VPC 中的 EC2 实例在 curl 中执行此操作会得到来自 ES 域的响应没有任何问题:

curl vpc-es-domain-AMAZON.eu-west-1.es.amazonaws.com:80

非常感谢帮助，因为我已经受困于此。

Answer 1

在您的设置中发现了两个问题

您只在堆栈中创建一个子网，并且只将一个子网分配给 Lambda。您需要为 Lambda
You need to fix access policy for ES。我手动更新到 "Don't require signing requst with IAM credential" 以便在控制台中进行测试。

我更新了 Cloudformation 模板以创建基于 python 的 lambda 处理程序以从同一 vpc 查询弹性搜索。它不完整，但如果您确定上述问题，那么它应该可以工作。

AWSTemplateFormatVersion: 2010-09-09
Description: The AWS CloudFormation tutorial
Resources:

  SomeDeploymentBucket:
    Type: 'AWS::S3::Bucket'

  AppLogGroup:
    Type: 'AWS::Logs::LogGroup'
    Properties:
      LogGroupName: /aws/lambda/some-lambda

    # ========= The Lambda Execution Role =========

  IamRoleLambdaExecution:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Policies:
        - PolicyName: !Join
            - '-'
            - - dev
              - some-app
              - lambda
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:*'
                  - 'rds-db:connect'
                  - 'rds:*'
                  - 'es:*'
                Resource: '*'


      Path: /
      RoleName: !Join
        - '-'
        - - some-app
          - dev
          - eu-west-1
          - lambdaRole
      ManagedPolicyArns:
        - !Join
          - ''
          - - 'arn:'
            - !Ref 'AWS::Partition'
            - ':iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'

    # ========= The Lambda =========

  AppLambdaFunction:
    Type: AWS::Lambda::Function
    DependsOn:
      - AppLogGroup
      - IamRoleLambdaExecution
    Properties:
      FunctionName: some-lambda
      Handler: index.lambda_handler
      Runtime: python2.7
      Timeout: 60
      MemorySize: 1024
      Role: !GetAtt
        - IamRoleLambdaExecution
        - Arn
      VpcConfig:
        SecurityGroupIds:
          - !Ref xxxVPCSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1
          - !Ref xxxLambdaSubnet2

      Code:
        ZipFile: !Sub |
          from __future__ import print_function
          import boto3
          iam = boto3.client('iam')

          def lambda_handler(event, context):
            print('called lambda_handler')

    # ========= VPC =========

  xxxVPC:
    Type: 'AWS::EC2::VPC'
    Properties:
      CidrBlock: 172.31.0.0/16
      InstanceTenancy: default
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'

  xxxVPCSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: VPC SG
      GroupDescription: VPC Security Group
      VpcId: !Ref xxxVPC

  xxxLambdaSubnet1:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.32.0/20
  xxxLambdaSubnet2:
    Type: 'AWS::EC2::Subnet'
    Properties:
      VpcId: !Ref xxxVPC
      CidrBlock: 172.31.16.0/20


    # ========= Elasticsearch =========

  xxxESSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupName: ES SG
      GroupDescription: ES Security group
      VpcId: !Ref xxxVPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: 0
          ToPort: 65535
          SourceSecurityGroupId: !Ref xxxVPCSecurityGroup

  xxxElasticSearch:
    Type: 'AWS::Elasticsearch::Domain'
    Properties:
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Action:
              - 'es:*'
              - 'ec2:*'
              - 's3:*'
            Principal:
              AWS:
                - '*'
            Resource: '*'
            Effect: Allow
      DomainName: es-xxx-domain
      EBSOptions:
        EBSEnabled: true
        Iops: 0
        VolumeSize: 20
        VolumeType: "gp2"
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'

      ElasticsearchClusterConfig:
        InstanceCount: 1
        InstanceType: m4.large.elasticsearch
        DedicatedMasterEnabled: 'false'
      VPCOptions:
        SecurityGroupIds:
          - !Ref xxxESSecurityGroup
        SubnetIds:
          - !Ref xxxLambdaSubnet1

更新了 Lambda 函数处理程序代码

  import urllib2

def lambda_handler(event, context):
    print('called lambda_handler')
    data = ''
    url = 'https://vpc-es-xxx-domain-fixthis.es.amazonaws.com'
    req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
    f = urllib2.urlopen(req)
    for x in f:
        print(x)
    f.close()

允许VPC中的Lambda访问同一VPC中的Elasticsearch域

Allowing Lambda in a VPC to access an Elasticsearch domain in the same VPC

amazon-ec2

amazon-web-services

elasticsearch

amazon-cloudformation

amazon-vpc