AWS 管理的 elasticsearch S3 快照权限

Question

我有一个 AWS 托管的 Elasticsearch 实例（较旧的“OpenSearch”之前的变体）。我想启用快照备份到 S3，但我似乎无法获得正确的权限。

我正在尝试使用实例角色，而不是特定的用户凭据。我试过关注 this guide 以及许多其他非常相似的人。

我有一个角色：arn:aws:iam::MYACCOUNTID:role/QA-elasticsearch-instance-role。该角色具有以下策略：

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListS3BackupBucket",
            "Action": [
                "iam:PassRole",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:iam::MYACCOUNTID:role/QA-elasticsearch-instance-role",
                "arn:aws:s3:::qa-mycompanyname-elasticsearch-backups"
            ]
        },
        {
            "Sid": "AllowBackupStorage",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::qa-mycompanyname-elasticsearch-backups/*"
            ]
        },
        {
            "Sid": "AllowElasticSearchPost",
            "Effect": "Allow",
            "Action": "es:ESHttpPost",
            "Resource": "arn:aws:es:us-east-1:MYACCOUNTID:domain/qa-elasticsearch"
        }
    ]
}

此角色还具有以下信任关系：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

我有以下 python 脚本来尝试创建快照存储库：

import boto3
import requests
from requests_aws4auth import AWS4Auth

host = 'https://MYELASTICSEARCHENDPOINT.us-east-1.es.amazonaws.com/'
region = 'us-east-1' # For example, us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()

awsauth = AWS4Auth("MYACCESSKEY", "MYSECRETKEY", region, service)

# Register repository
path = '_snapshot/qa_snapshot_repository' # the Elasticsearch API endpoint
url = host + path

payload = {
    "type": "s3",
    "settings": {
        "bucket": "qa-MYCOMPANYNAME-elasticsearch-backups",
        "region": "us-east-1",
        "role_arn": "arn:aws:iam::MYACCOUNTID:role/QA-elasticsearch-instance-role"
    }
}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)
print(r.text)

当我运行这个时，(python3 elasticsearch_test.py | jq .) 我得到以下信息：

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[qa_snapshot_repository] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[qa_snapshot_repository] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-4-VcUFXhTcW4PQ6fi6AiVA/master.dat] using a single upload",
      "caused_by": {
        "type": "amazon_s3_exception",
        "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: RJKPNBBEQQBNWHWT; S3 Extended Request ID: 3O5AqSkKhsCNlmYLCtcfnE0omhp6wdbPCejJ2r4dNJq93l2xppSs4Hv6uTGvNIJ2jFWLt31gsyc=)"
      }
    }
  },
  "status": 500
}

我已经用谷歌搜索并尝试调整权限几个小时了，但似乎无法正确调整。有什么明显的我想念的吗？

Answer 1

所以事实证明错误响应具有超级误导性。这并不是真正的“AccessDenied”问题，而是由于客户端和服务器之间的加密不正确而导致对 S3 的请求被拒绝。在 this post 之后，我决定尝试添加 "server_side_encryption": "true"，一切正常。

payload = {
    "type": "s3",
    "settings": {
        "bucket": "qa-mycompanyname-elasticsearch-backups",
        "region": "us-east-1",
        "role_arn": "arn:aws:iam::myaccountid:role/QA-elasticsearch-instance-role",
        "server_side_encryption": "true"
    }
}

200
{
  "acknowledged": true
}

我无法正确解释为什么会这样。我猜它告诉 Elasticsearch 实例上的客户端使用 https URL 与 S3 通信而不是 http，但似乎疯狂的是 http 将是默认值，或者没有更好的错误回消息。

AWS 管理的 elasticsearch S3 快照权限

AWS managed elasticsearch S3 snapshot permissions

elasticsearch

amazon-s3

aws-elasticsearch