通过 Cloud Formation 创建 Amazon Elasticsearch Service 时 CloudWatch 资源访问策略错误
CloudWatch resource access policy error while creating Amazon Elasticsearch Service via Cloud Formation
-
amazon-web-services
-
elasticsearch
-
amazon-cloudformation
-
amazon-iam
-
aws-cloudformation-custom-resource
我正在尝试创建一个启用 LogPublishingOptions 的弹性搜索域。启用 LogPublishingOptions ES 时表示它没有足够的权限在 Cloudwatch 上创建 LogStream。
我尝试创建一个具有角色的策略并将该策略附加到 ES 引用的日志组，但它不起作用。以下是我的elastic search cloud formation模板，
AWSTemplateFormatVersion: 2010-09-09
Resources:
MYLOGGROUP:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: index_slow
MYESROLE:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: es.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AmazonESFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
RoleName: !Join
- '-'
- - es
- !Ref 'AWS::Region'
PolicyDocESIndexSlow :
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:PutLogEvents
- logs:CreateLogStream
Resource: 'arn:aws:logs:*'
PolicyName: !Ref MYLOGGROUP
Roles:
- !Ref MYESROLE
MYESDOMAIN:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: 'es-domain'
ElasticsearchVersion: '7.4'
ElasticsearchClusterConfig:
DedicatedMasterCount: 3
DedicatedMasterEnabled: True
DedicatedMasterType: 'r5.large.elasticsearch'
InstanceCount: '2'
InstanceType: 'r5.large.elasticsearch'
EBSOptions:
EBSEnabled: True
VolumeSize: 10
VolumeType: 'gp2'
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: '*'
Action: 'es:*'
Resource: '*'
AdvancedOptions:
rest.action.multi.allow_explicit_index: True
LogPublishingOptions:
INDEX_SLOW_LOGS:
CloudWatchLogsLogGroupArn: !GetAtt
- MYLOGGROUP
- Arn
Enabled: True
VPCOptions:
SubnetIds:
- !Ref MYSUBNET
SecurityGroupIds:
- !Ref MYSECURITYGROUP
MYVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
MYSUBNET:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref MYVPC
CidrBlock: 10.0.0.0/16
MYSECURITYGROUP:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: security group for elastic search domain
VpcId: !Ref MYVPC
GroupName: 'SG for ES'
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
执行后，它会创建除 MYESDOMAIN 之外的所有资源。它说
为 CloudWatch Logs 日志组 index_slow 指定的资源访问策略未授予 Amazon Elasticsearch Service 创建日志流的足够权限。请检查资源访问策略。 （服务：AWSElasticsearch；状态代码：400；错误代码：ValidationException）
知道这里缺少什么吗？
我认为这里有些混淆应该updated/set 启用 ES 写入日志组的策略。
我认为您应该将 PolicyDocESIndexSlow 策略应用到 CloudWatch Logs。
根据我的记忆，这 无法在 CloudFormation 中完成。您必须使用 put-resource-policy、相应的 API 调用或控制台，如下所示：
最终代码应该是这样的，
部署ES lambda_function.py
import logging
import time
import boto3
import json
from crhelper import CfnResource
logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)
try:
# Init code goes here
pass
except Exception as e:
helper.init_failure(e)
@helper.create
@helper.update
def create(event, _):
logger.info("Got Create/Update")
my_log_group_arn = event['ResourceProperties']['MYLOGGROUPArn']
client = boto3.client('logs')
policy_document = dict()
policy_document['Version'] = '2012-10-17'
policy_document['Statement'] = [{
'Sid': 'ESLogsToCloudWatchLogs',
'Effect': 'Allow',
'Principal': {
'Service': [
'es.amazonaws.com'
]
},
'Action': 'logs:*',
}]
policy_document['Statement'][0]['Resource'] = my_log_group_arn
client.put_resource_policy(policyName='ESIndexSlowPolicy', policyDocument=json.dumps(policy_document))
helper.Data['success'] = True
helper.Data['message'] = 'ES policy deployment successful'
# To return an error to Cloud Formation you raise an exception:
if not helper.Data["success"]:
raise Exception('Error message to cloud formation')
return "MYESIDDEFAULT"
@helper.delete
def delete(event, _):
logger.info("Got Delete")
# Delete never returns anything. Should not fail if the underlying resources are already deleted.
# Desired state.
try:
client = boto3.client('logs')
client.delete_resource_policy(policyName='ESIndexSlowPolicy')
except Exception as ex:
logger.critical(f'ES policy delete failed with error [{repr(ex)}]')
def lambda_handler(event, context):
helper(event, context)
以及 CF 模板中的一些附加组件
MYLAMBDAROLE:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
- 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
- 'arn:aws:iam::aws:policy/AmazonESFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
RoleName: !Join
- '-'
- - lambda-role
- !Ref 'AWS::Region'
MYLAMBDADEPLOY:
Type: 'AWS::Lambda::Function'
Properties:
Code:
S3Bucket: es-bucket-for-lambda-ta86asdf596
S3Key: es.zip
FunctionName: deploy_es
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt
- MYLAMBDAROLE
- Arn
Runtime: python3.8
Timeout: 60
MYESSETUP:
Type: 'Custom::MYESSETUP'
Properties:
ServiceToken: !GetAtt
- MYLAMBDADEPLOY
- Arn
MYLOGGROUPArn: !GetAtt
- MYLOGGROUP
- Arn
DependsOn:
- MYLAMBDADEPLOY
- MYLOGGROUP
然后只需将 DependsOn 下方添加到 MYESDOMAIN
DependsOn:
- MYESSETUP
2021 年更新
有一个名为 AWS::Logs::ResourcePolicy 的 CloudFormation 资源，它允许在 CF 中为 CloudWatch Logs 定义策略。我发现的主要问题是它只接受一个真正的字符串作为值。尝试使用 Ref、Join 等 assemble 字符串一直被拒绝。如果有人能完成这项工作，那就太棒了。
用 YAML 编写更容易，因为 JSON 需要转义所有 " 个字符。
OSLogGroupPolicy:
Type: AWS::Logs::ResourcePolicy
Properties:
PolicyName: AllowES
PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'
amazon-web-services
elasticsearch
amazon-cloudformation
amazon-iam
aws-cloudformation-custom-resource
我正在尝试创建一个启用 LogPublishingOptions 的弹性搜索域。启用 LogPublishingOptions ES 时表示它没有足够的权限在 Cloudwatch 上创建 LogStream。
我尝试创建一个具有角色的策略并将该策略附加到 ES 引用的日志组，但它不起作用。以下是我的elastic search cloud formation模板，
AWSTemplateFormatVersion: 2010-09-09
Resources:
MYLOGGROUP:
Type: 'AWS::Logs::LogGroup'
Properties:
LogGroupName: index_slow
MYESROLE:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: es.amazonaws.com
Action: 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AmazonESFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
RoleName: !Join
- '-'
- - es
- !Ref 'AWS::Region'
PolicyDocESIndexSlow :
Type: AWS::IAM::Policy
Properties:
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- logs:PutLogEvents
- logs:CreateLogStream
Resource: 'arn:aws:logs:*'
PolicyName: !Ref MYLOGGROUP
Roles:
- !Ref MYESROLE
MYESDOMAIN:
Type: AWS::Elasticsearch::Domain
Properties:
DomainName: 'es-domain'
ElasticsearchVersion: '7.4'
ElasticsearchClusterConfig:
DedicatedMasterCount: 3
DedicatedMasterEnabled: True
DedicatedMasterType: 'r5.large.elasticsearch'
InstanceCount: '2'
InstanceType: 'r5.large.elasticsearch'
EBSOptions:
EBSEnabled: True
VolumeSize: 10
VolumeType: 'gp2'
AccessPolicies:
Version: 2012-10-17
Statement:
- Effect: Deny
Principal:
AWS: '*'
Action: 'es:*'
Resource: '*'
AdvancedOptions:
rest.action.multi.allow_explicit_index: True
LogPublishingOptions:
INDEX_SLOW_LOGS:
CloudWatchLogsLogGroupArn: !GetAtt
- MYLOGGROUP
- Arn
Enabled: True
VPCOptions:
SubnetIds:
- !Ref MYSUBNET
SecurityGroupIds:
- !Ref MYSECURITYGROUP
MYVPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: 10.0.0.0/16
MYSUBNET:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref MYVPC
CidrBlock: 10.0.0.0/16
MYSECURITYGROUP:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: security group for elastic search domain
VpcId: !Ref MYVPC
GroupName: 'SG for ES'
SecurityGroupIngress:
- FromPort: '443'
IpProtocol: tcp
ToPort: '443'
CidrIp: 0.0.0.0/0
执行后，它会创建除 MYESDOMAIN 之外的所有资源。它说
为 CloudWatch Logs 日志组 index_slow 指定的资源访问策略未授予 Amazon Elasticsearch Service 创建日志流的足够权限。请检查资源访问策略。 （服务：AWSElasticsearch；状态代码：400；错误代码：ValidationException）
知道这里缺少什么吗？
我认为这里有些混淆应该updated/set 启用 ES 写入日志组的策略。
我认为您应该将 PolicyDocESIndexSlow 策略应用到 CloudWatch Logs。
根据我的记忆，这 无法在 CloudFormation 中完成。您必须使用 put-resource-policy、相应的 API 调用或控制台，如下所示：
最终代码应该是这样的，
部署ES lambda_function.py
import logging
import time
import boto3
import json
from crhelper import CfnResource
logger = logging.getLogger(__name__)
helper = CfnResource(json_logging=False, log_level='DEBUG', boto_level='CRITICAL', sleep_on_delete=120)
try:
# Init code goes here
pass
except Exception as e:
helper.init_failure(e)
@helper.create
@helper.update
def create(event, _):
logger.info("Got Create/Update")
my_log_group_arn = event['ResourceProperties']['MYLOGGROUPArn']
client = boto3.client('logs')
policy_document = dict()
policy_document['Version'] = '2012-10-17'
policy_document['Statement'] = [{
'Sid': 'ESLogsToCloudWatchLogs',
'Effect': 'Allow',
'Principal': {
'Service': [
'es.amazonaws.com'
]
},
'Action': 'logs:*',
}]
policy_document['Statement'][0]['Resource'] = my_log_group_arn
client.put_resource_policy(policyName='ESIndexSlowPolicy', policyDocument=json.dumps(policy_document))
helper.Data['success'] = True
helper.Data['message'] = 'ES policy deployment successful'
# To return an error to Cloud Formation you raise an exception:
if not helper.Data["success"]:
raise Exception('Error message to cloud formation')
return "MYESIDDEFAULT"
@helper.delete
def delete(event, _):
logger.info("Got Delete")
# Delete never returns anything. Should not fail if the underlying resources are already deleted.
# Desired state.
try:
client = boto3.client('logs')
client.delete_resource_policy(policyName='ESIndexSlowPolicy')
except Exception as ex:
logger.critical(f'ES policy delete failed with error [{repr(ex)}]')
def lambda_handler(event, context):
helper(event, context)
以及 CF 模板中的一些附加组件
MYLAMBDAROLE:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/AWSLambdaFullAccess'
- 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
- 'arn:aws:iam::aws:policy/AmazonESFullAccess'
- 'arn:aws:iam::aws:policy/CloudWatchFullAccess'
RoleName: !Join
- '-'
- - lambda-role
- !Ref 'AWS::Region'
MYLAMBDADEPLOY:
Type: 'AWS::Lambda::Function'
Properties:
Code:
S3Bucket: es-bucket-for-lambda-ta86asdf596
S3Key: es.zip
FunctionName: deploy_es
Handler: lambda_function.lambda_handler
MemorySize: 128
Role: !GetAtt
- MYLAMBDAROLE
- Arn
Runtime: python3.8
Timeout: 60
MYESSETUP:
Type: 'Custom::MYESSETUP'
Properties:
ServiceToken: !GetAtt
- MYLAMBDADEPLOY
- Arn
MYLOGGROUPArn: !GetAtt
- MYLOGGROUP
- Arn
DependsOn:
- MYLAMBDADEPLOY
- MYLOGGROUP
然后只需将 DependsOn 下方添加到 MYESDOMAIN
DependsOn:
- MYESSETUP
2021 年更新
有一个名为 AWS::Logs::ResourcePolicy 的 CloudFormation 资源，它允许在 CF 中为 CloudWatch Logs 定义策略。我发现的主要问题是它只接受一个真正的字符串作为值。尝试使用 Ref、Join 等 assemble 字符串一直被拒绝。如果有人能完成这项工作，那就太棒了。
用 YAML 编写更容易，因为 JSON 需要转义所有 " 个字符。
OSLogGroupPolicy:
Type: AWS::Logs::ResourcePolicy
Properties:
PolicyName: AllowES
PolicyDocument: '{"Version": "2012-10-17","Statement":[{"Effect":"Allow","Principal": {"Service": ["es.amazonaws.com"]},"Action":["logs:PutLogEvents","logs:CreateLogStream"],"Resource":"*"}]}'