尽管有 SubscriptionFilter，AWS CloudWatch LogGroup 从不向 Lambda 发送日志

Question

Objective：设置一个以日志组名称作为参数的 Cloudformation 堆栈，每当该日志组中出现新日志时，它们就会被发送到 Lambda 函数进行处理，然后发送到 Kinesis Firehose，然后将日志文件发送到名为 foobarbaz 的存储桶。

问题：Lambda 函数永远不会被调用（Lambda 的 CloudWatch 日志显示即使在日志组中出现新数据后它也永远不会被触发）。由于我设置了 SubscriptionFilter 资源，调用应该会自动发生。我没有看到任何错误。无论发生什么，似乎都在默默地失败。

注意：SubscriptionFilter 上的 FilterPattern 已设置为空字符串。我这样做的目的是将所有日志从日志组发送到 Lambda 函数。

这是我的 Cloudformation 模板：

Parameters:
  LogGroupName:
    Type: String
    Description: The name of the log group who's logs we want to send to send to Lambda->Kinesis->S3

  AuditTrailPrefix:
    Type: String
    Description: Log files will be sent to the Logging account S3 bucket with this prefix in the bucket path

Resources:  
  AuditTrailFunctionPermissions:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !Ref AuditTrailFunction
      Principal: logs.amazonaws.com
      SourceAccount: !Ref AWS::AccountId

  AuditTrailFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role: !GetAtt AuditTrailFunctionRole.Arn
      Code:
        ZipFile: >
          // do some stuff with the data and PUT it to KinesisFirehose
          // removed for brevity
      Runtime: nodejs8.10
      Timeout: 30

  AuditTrailFunctionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
        Version: '2012-10-17'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyDocument:
            Statement:
              - Action:
                  - firehose:PutRecord
                  - firehose:PutRecordBatch
                Effect: Allow
                Resource: !Sub arn:aws:firehose:${AWS::Region}:${AWS::AccountId}:deliverystream/${AuditTrailDeliveryStream}
            Version: '2012-10-17'
          PolicyName: root

  AuditTrailSubscription:
    Type: AWS::Logs::SubscriptionFilter
    DependsOn: AuditTrailFunctionPermissions
    Properties:
      DestinationArn: !GetAtt AuditTrailFunction.Arn
      FilterPattern: ''
      LogGroupName: !Ref LogGroupName

  AuditTrailDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamType: DirectPut
      S3DestinationConfiguration:
        BucketARN: arn:aws:s3:::foobarbaz
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 50
        CompressionFormat: GZIP
        Prefix: !Ref AuditTrailPrefix
        RoleARN: !GetAtt DeliveryRole.Arn

  DeliveryRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          Effect: Allow
          Principal:
            Service: firehose.amazonaws.com
          Action: sts:AssumeRole

      Policies:
        - PolicyName: firehose_delivery_policy
          PolicyDocument:
            Statement:
              Effect: Allow
              Action:
                - s3:AbortMultipartUpload
                - s3:GetBucketLocation
                - s3:GetObject
                - s3:ListBucket
                - s3:ListBucketMultipartUploads
                - s3:PutObject
              Resource:
                - arn:aws:s3:::foobarbaz
                - arn:aws:s3:::foobarbaz/${AuditTrailPrefix}*

Answer 1

我看不出有什么问题，但这里有一些故障排除提示：

只有当新日志上传到您的日志组时，才会调用 Lambda 函数。在您设置订阅过滤器之前，它不会为日志组中已有的数据调用。
如果 #1 不是这种情况（即您有新数据正在上传），请转至 CloudWatch -> Metrics 并搜索日志组名称。您应该找到 4 个与您的订阅过滤器相关的指标：ForwardedBytes、ForwardedLogEvents、DeliveryErrors、DeliveryThrottling。有关说明，请参阅 this。如果 DeliveryErrors 或 DeliveryThrottling 指标 > 0，则说明存在问题。
DeliveryErrors 最有可能的问题是权限问题。我看不出你的有什么问题，但我会先仔细检查一下。
您可以使用 AWS CLI 手动调试订阅设置。（参见 this。）这可以帮助您找出可能导致问题的设置部分。

Answer 2

我怀疑这是因为您的 AuditTrailSubscription 资源中缺少 RoleArn。没有它，Cloudwatch 没有执行您的 Lambda 函数的权限。

这是 AWS::Logs::SubscriptionFilter 的文档页面 https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-logs-subscriptionfilter.html

尽管有 SubscriptionFilter，AWS CloudWatch LogGroup 从不向 Lambda 发送日志

AWS CloudWatch LogGroup never sending logs to Lambda despite SubscriptionFilter

amazon-cloudformation

amazon-cloudwatch

amazon-kinesis

aws-lambda

amazon-kinesis-firehose