您能否设置从 S3 存储桶子文件夹到 GCS 存储桶的传输，仅使用访问 S3 子文件夹而不是根文件夹的凭据？

Question

我正在寻找一个传输作业来获取存储在 S3 存储桶中的文件并将它们加载到 GCS 存储桶中。我拥有的凭据允许我访问包含我需要的 S3 文件的文件夹，但不能访问更高级别的文件夹。

当我尝试使用 'Amazon S3 bucket' 下的 S3 存储桶名称设置传输作业并填写访问密钥 ID 和秘密访问密钥时，访问被拒绝，正如您所期望的那样，给定的限制我的凭据。但是，如果我将额外的路径信息添加为前缀项（例如 'Production/FTP/CompanyName'）并且我确实可以访问此文件夹，则访问仍然被拒绝。

似乎我无法克服我无法访问根目录的事实。有什么解决办法吗？

Answer 1

保罗，

很可能您的 IAM 角色缺少 s3:ListBucket 权限。您可以将您的 IAM 角色更新为 s3:ListBucket 、 s3:GetBucketLocation 并重试吗？

Answer 2

根据文档link：

The Storage Transfer Service uses the project-[$PROJECT_NUMBER]@storage-transfer-service.iam.gserviceaccount.com service account to move data from a Cloud Storage source bucket.

The service account must have the following permissions for the source bucket:

storage.buckets.get Allows the service account to get the location of the bucket. Always required.

storage.objects.list Allows the service account to list objects in the bucket. Always required.

storage.objects.get Allows the service account to read objects in the bucket. Always required.

storage.objects.delete Allows the service account to delete objects in the bucket. Required if you set deleteObjectsFromSourceAfterTransfer to true.

The roles/storage.objectViewer and roles/storage.legacyBucketReader roles together contain the permissions that are always required. The roles/storage.legacyBucketWriter role contains the storage.objects.delete permissions. The service account used to perform the transfer must be assigned the desired roles.

您必须在您的 AWS 存储桶上设置此权限。

Answer 3

在 AWS 上的权限策略应该如下所示，以防您想授予对子文件夹的访问权限。

{
    "Version": "2012-10-17",

    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<bucketname>",
                "arn:aws:s3:::<bucketname>/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": "arn:aws:s3:::<bucketname>",
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "<subfolder>/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*"
            ],
            "Resource": [
                "arn:aws:s3:::<bucketname>/<subfolder>",
                "arn:aws:s3:::<bucketname>/<subfolder>/*"
            ],
            "Condition": {}
        }
    ]
}

您能否设置从 S3 存储桶子文件夹到 GCS 存储桶的传输，仅使用访问 S3 子文件夹而不是根文件夹的凭据？

Can you set up a transfer from an S3 bucket subfolder to a GCS bucket with only the credentials to access the S3 subfolder, not the root folder?

data-transfer

amazon-s3

subdirectory

google-cloud-storage

google-cloud-platform