Terraform:配置 cloudwatch 日志订阅交付到 lambda?
Terraform: configuring cloudwatch log subscription delivery to lambda?
我需要将我的 cloudwatch 日志发送到日志分析服务。
我已经阅读了这些文章 here and here 并手动完成了它,不用担心。
现在我正在尝试使用 Terraform(roles/policies、安全组、cloudwatch 日志组、lambda 以及从日志组触发 lambda 来自动执行所有这些操作。
但我不知道如何使用 TF 配置 AWS 以从 cloudwatch 日志中触发 lambda。
我可以通过执行以下操作(在 Lambda Web 控制台 UI 中)手动 link 两个 TF 资源:
- 进入 lambda 函数的 "Triggers" 部分
- 点击"Add Trigger"
- select "cloudwatch logs" 来自触发器类型列表
- select我要触发lambda的日志组
- 输入过滤器名称
- 将过滤器模式留空(暗示在所有日志流上触发)
- 确保 "enable trigger" 是 selected
- 点击提交按钮
完成后,lambda 将显示在 cloudwatch 日志控制台的订阅列中 - 显示为 "Lambda (cloudwatch-sumologic-lambda)"。
我尝试使用以下 TF 资源创建订阅:
resource "aws_cloudwatch_log_subscription_filter" "cloudwatch-sumologic-lambda-subscription" {
name = "cloudwatch-sumologic-lambda-subscription"
role_arn = "${aws_iam_role.jordi-waf-cloudwatch-lambda-role.arn}"
log_group_name = "${aws_cloudwatch_log_group.jordi-waf-int-app-loggroup.name}"
filter_pattern = "logtype test"
destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}
但它失败了:
aws_cloudwatch_log_subscription_filter.cloudwatch-sumologic-lambda-subscription: InvalidParameterException: destinationArn for vendor lambda cannot be used with roleArn
我发现 关于为预定事件设置类似的东西,但这似乎不等同于我上面描述的控制台操作(控制台 UI 方法没有创建我可以看到的 event/rule)。
有人可以指点我做错了什么吗?
我的 aws_cloudwatch_log_subscription_filter
资源定义不正确 - 在这种情况下你不应该提供 role_arn
参数。
您还需要添加一个 aws_lambda_permission
资源(在过滤器上定义了一个 depends_on
关系,否则 TF 可能会以错误的顺序执行)。
请注意,AWS lambda 控制台 UI 为您添加了无形的 lambda 权限,因此请注意,如果您之前碰巧做过相同的操作,aws_cloudwatch_log_subscription_filter
将在没有权限资源的情况下工作控制台 UI.
必要的 TF 配置如下所示(最后两个资源是配置实际 cloudwatch->lambda
触发器的相关资源):
// intended for application logs (access logs, modsec, etc.)
resource "aws_cloudwatch_log_group" "test-app-loggroup" {
name = "test-app"
retention_in_days = 90
}
resource "aws_security_group" "cloudwatch-sumologic-lambda-sg" {
name = "cloudwatch-sumologic-lambda-sg"
tags {
Name = "cloudwatch-sumologic-lambda-sg"
}
description = "Security group for lambda to move logs from CWL to SumoLogic"
vpc_id = "${aws_vpc.dev-vpc.id}"
}
resource "aws_security_group_rule" "https-egress-cloudwatch-sumologic-to-internet" {
type = "egress"
from_port = 443
to_port = 443
protocol = "tcp"
security_group_id = "${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"
cidr_blocks = ["0.0.0.0/0"]
}
resource "aws_iam_role" "test-cloudwatch-lambda-role" {
name = "test-cloudwatch-lambda-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
EOF
}
resource "aws_iam_role_policy" "test-cloudwatch-lambda-policy" {
name = "test-cloudwatch-lambda-policy"
role = "${aws_iam_role.test-cloudwatch-lambda-role.id}"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole1",
"Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface"
],
"Resource": "*"
},
{
"Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole2",
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "arn:aws:ec2:ap-southeast-2:${var.dev_vpc_account_id}:network-interface/*"
},
{
"Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole1",
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:*"
},
{
"Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole2",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:log-group:/aws/lambda/*"
]
},
{
"Sid": "CopiedFromTemplateAWSLambdaAMIExecutionRole",
"Effect": "Allow",
"Action": [
"ec2:DescribeImages"
],
"Resource": "*"
}
]
}
EOF
}
resource "aws_lambda_function" "cloudwatch-sumologic-lambda" {
function_name = "cloudwatch-sumologic-lambda"
filename = "${var.lambda_dir}/cloudwatchSumologicLambda.zip"
source_code_hash = "${base64sha256(file("${var.lambda_dir}/cloudwatchSumologicLambda.zip"))}"
handler = "cloudwatchSumologic.handler"
role = "${aws_iam_role.test-cloudwatch-lambda-role.arn}"
memory_size = "128"
runtime = "nodejs4.3"
// set low because I'm concerned about cost-blowout in the case of mis-configuration
timeout = "15"
vpc_config = {
subnet_ids = ["${aws_subnet.dev-private-subnet.id}"]
security_group_ids = ["${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"]
}
}
resource "aws_lambda_permission" "test-app-allow-cloudwatch" {
statement_id = "test-app-allow-cloudwatch"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
principal = "logs.ap-southeast-2.amazonaws.com"
source_arn = "${aws_cloudwatch_log_group.test-app-loggroup.arn}"
}
resource "aws_cloudwatch_log_subscription_filter" "test-app-cloudwatch-sumologic-lambda-subscription" {
depends_on = ["aws_lambda_permission.test-app-allow-cloudwatch"]
name = "cloudwatch-sumologic-lambda-subscription"
log_group_name = "${aws_cloudwatch_log_group.test-app-loggroup.name}"
filter_pattern = ""
destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}
编辑:请注意上面的 TF 代码是几年前写的,使用版本 0.11.x
- 它应该仍然有效,但可能有更好的做事方式.具体来说,除非需要,否则不要使用这样的内联策略,而是使用 aws_iam_policy_document - 随着时间的推移,它们更容易维护。
使用 Terraform v0.12.29
和 AWS 提供商 v3.1.0
我遇到了一个奇怪的问题,这让我花了几个小时进行调试。
为了节省其他人一些宝贵的时间,我将分享它作为对已接受答案的补充。
cloudwatch日志组的值arn:
${aws_cloudwatch_log_group.test-app-loggroup.arn}
未正确插值 - 输出末尾缺少一个“:*
”。
这会导致以下错误:
Error creating {the-calling-service}: InvalidCloudWatchLogsLogGroupArnException:
Check the log group ARN: {the-calling-service} can't validate it.
添加 :*
后缀解决了问题:
source_arn = "${aws_cloudwatch_log_group.test-app-loggroup.arn}:*" #<----Notice the :* postfix
我需要将我的 cloudwatch 日志发送到日志分析服务。
我已经阅读了这些文章 here and here 并手动完成了它,不用担心。
现在我正在尝试使用 Terraform(roles/policies、安全组、cloudwatch 日志组、lambda 以及从日志组触发 lambda 来自动执行所有这些操作。
但我不知道如何使用 TF 配置 AWS 以从 cloudwatch 日志中触发 lambda。
我可以通过执行以下操作(在 Lambda Web 控制台 UI 中)手动 link 两个 TF 资源:
- 进入 lambda 函数的 "Triggers" 部分
- 点击"Add Trigger"
- select "cloudwatch logs" 来自触发器类型列表
- select我要触发lambda的日志组
- 输入过滤器名称
- 将过滤器模式留空(暗示在所有日志流上触发)
- 确保 "enable trigger" 是 selected
- 点击提交按钮
完成后,lambda 将显示在 cloudwatch 日志控制台的订阅列中 - 显示为 "Lambda (cloudwatch-sumologic-lambda)"。
我尝试使用以下 TF 资源创建订阅:
resource "aws_cloudwatch_log_subscription_filter" "cloudwatch-sumologic-lambda-subscription" {
name = "cloudwatch-sumologic-lambda-subscription"
role_arn = "${aws_iam_role.jordi-waf-cloudwatch-lambda-role.arn}"
log_group_name = "${aws_cloudwatch_log_group.jordi-waf-int-app-loggroup.name}"
filter_pattern = "logtype test"
destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}
但它失败了:
aws_cloudwatch_log_subscription_filter.cloudwatch-sumologic-lambda-subscription: InvalidParameterException: destinationArn for vendor lambda cannot be used with roleArn
我发现
有人可以指点我做错了什么吗?
我的 aws_cloudwatch_log_subscription_filter
资源定义不正确 - 在这种情况下你不应该提供 role_arn
参数。
您还需要添加一个 aws_lambda_permission
资源(在过滤器上定义了一个 depends_on
关系,否则 TF 可能会以错误的顺序执行)。
请注意,AWS lambda 控制台 UI 为您添加了无形的 lambda 权限,因此请注意,如果您之前碰巧做过相同的操作,aws_cloudwatch_log_subscription_filter
将在没有权限资源的情况下工作控制台 UI.
必要的 TF 配置如下所示(最后两个资源是配置实际 cloudwatch->lambda
触发器的相关资源):
// intended for application logs (access logs, modsec, etc.)
resource "aws_cloudwatch_log_group" "test-app-loggroup" {
name = "test-app"
retention_in_days = 90
}
resource "aws_security_group" "cloudwatch-sumologic-lambda-sg" {
name = "cloudwatch-sumologic-lambda-sg"
tags {
Name = "cloudwatch-sumologic-lambda-sg"
}
description = "Security group for lambda to move logs from CWL to SumoLogic"
vpc_id = "${aws_vpc.dev-vpc.id}"
}
resource "aws_security_group_rule" "https-egress-cloudwatch-sumologic-to-internet" {
type = "egress"
from_port = 443
to_port = 443
protocol = "tcp"
security_group_id = "${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"
cidr_blocks = ["0.0.0.0/0"]
}
resource "aws_iam_role" "test-cloudwatch-lambda-role" {
name = "test-cloudwatch-lambda-role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Effect": "Allow"
}
]
}
EOF
}
resource "aws_iam_role_policy" "test-cloudwatch-lambda-policy" {
name = "test-cloudwatch-lambda-policy"
role = "${aws_iam_role.test-cloudwatch-lambda-role.id}"
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole1",
"Effect": "Allow",
"Action": [
"ec2:CreateNetworkInterface"
],
"Resource": "*"
},
{
"Sid": "CopiedFromTemplateAWSLambdaVPCAccessExecutionRole2",
"Effect": "Allow",
"Action": [
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "arn:aws:ec2:ap-southeast-2:${var.dev_vpc_account_id}:network-interface/*"
},
{
"Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole1",
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:*"
},
{
"Sid": "CopiedFromTemplateAWSLambdaBasicExecutionRole2",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:ap-southeast-2:${var.dev_vpc_account_id}:log-group:/aws/lambda/*"
]
},
{
"Sid": "CopiedFromTemplateAWSLambdaAMIExecutionRole",
"Effect": "Allow",
"Action": [
"ec2:DescribeImages"
],
"Resource": "*"
}
]
}
EOF
}
resource "aws_lambda_function" "cloudwatch-sumologic-lambda" {
function_name = "cloudwatch-sumologic-lambda"
filename = "${var.lambda_dir}/cloudwatchSumologicLambda.zip"
source_code_hash = "${base64sha256(file("${var.lambda_dir}/cloudwatchSumologicLambda.zip"))}"
handler = "cloudwatchSumologic.handler"
role = "${aws_iam_role.test-cloudwatch-lambda-role.arn}"
memory_size = "128"
runtime = "nodejs4.3"
// set low because I'm concerned about cost-blowout in the case of mis-configuration
timeout = "15"
vpc_config = {
subnet_ids = ["${aws_subnet.dev-private-subnet.id}"]
security_group_ids = ["${aws_security_group.cloudwatch-sumologic-lambda-sg.id}"]
}
}
resource "aws_lambda_permission" "test-app-allow-cloudwatch" {
statement_id = "test-app-allow-cloudwatch"
action = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
principal = "logs.ap-southeast-2.amazonaws.com"
source_arn = "${aws_cloudwatch_log_group.test-app-loggroup.arn}"
}
resource "aws_cloudwatch_log_subscription_filter" "test-app-cloudwatch-sumologic-lambda-subscription" {
depends_on = ["aws_lambda_permission.test-app-allow-cloudwatch"]
name = "cloudwatch-sumologic-lambda-subscription"
log_group_name = "${aws_cloudwatch_log_group.test-app-loggroup.name}"
filter_pattern = ""
destination_arn = "${aws_lambda_function.cloudwatch-sumologic-lambda.arn}"
}
编辑:请注意上面的 TF 代码是几年前写的,使用版本 0.11.x
- 它应该仍然有效,但可能有更好的做事方式.具体来说,除非需要,否则不要使用这样的内联策略,而是使用 aws_iam_policy_document - 随着时间的推移,它们更容易维护。
使用 Terraform v0.12.29
和 AWS 提供商 v3.1.0
我遇到了一个奇怪的问题,这让我花了几个小时进行调试。
为了节省其他人一些宝贵的时间,我将分享它作为对已接受答案的补充。
cloudwatch日志组的值arn:
${aws_cloudwatch_log_group.test-app-loggroup.arn}
未正确插值 - 输出末尾缺少一个“:*
”。
这会导致以下错误:
Error creating {the-calling-service}: InvalidCloudWatchLogsLogGroupArnException: Check the log group ARN: {the-calling-service} can't validate it.
添加 :*
后缀解决了问题:
source_arn = "${aws_cloudwatch_log_group.test-app-loggroup.arn}:*" #<----Notice the :* postfix