PERMISSION_DENIED 从 S3 转移到 GCS

Question

我是GCS的新手，希望能很好地解释我的问题。我有 S3 用于将我的 Aperture/Photos 从我的 Macbook 备份到云端。我想将该数据迁移到 GCS 以便于使用、更好的界面、gsutil 等

我使用了 GCS 网站上的传输数据实用程序，并输入了我的 S3 凭据（访问密钥和秘密密钥）。我启用了互操作性，并在 GCS 中生成了一个密钥，尽管我认为这没有必要。

在 S3 中，我在 IAM (GoogleTransfer) 中创建了一个用户，并为该用户创建了一个内联策略，并选择了除以 Delete* 开头的选项之外的所有选项。这些权限是：

{
"Version": "2012-10-17",
"Statement": [
    {
        "Sid": "<my ID>",
        "Effect": "Allow",
        "Action": [
            "s3:AbortMultipartUpload",
            "s3:CreateBucket",
            "s3:GetAccelerateConfiguration",
            "s3:GetBucketAcl",
            "s3:GetBucketCORS",
            "s3:GetBucketLocation",
            "s3:GetBucketLogging",
            "s3:GetBucketNotification",
            "s3:GetBucketPolicy",
            "s3:GetBucketRequestPayment",
            "s3:GetBucketTagging",
            "s3:GetBucketVersioning",
            "s3:GetBucketWebsite",
            "s3:GetLifecycleConfiguration",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:GetObjectTorrent",
            "s3:GetObjectVersion",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTorrent",
            "s3:GetReplicationConfiguration",
            "s3:ListAllMyBuckets",
            "s3:ListBucket",
            "s3:ListBucketMultipartUploads",
            "s3:ListBucketVersions",
            "s3:ListMultipartUploadParts",
            "s3:PutAccelerateConfiguration",
            "s3:PutBucketAcl",
            "s3:PutBucketCORS",
            "s3:PutBucketLogging",
            "s3:PutBucketNotification",
            "s3:PutBucketPolicy",
            "s3:PutBucketRequestPayment",
            "s3:PutBucketTagging",
            "s3:PutBucketVersioning",
            "s3:PutBucketWebsite",
            "s3:PutLifecycleConfiguration",
            "s3:PutReplicationConfiguration",
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:PutObjectVersionAcl",
            "s3:ReplicateDelete",
            "s3:ReplicateObject",
            "s3:RestoreObject"
        ],
        "Resource": [
            "arn:aws:s3:::*"
        ]
    }
]

}

我在 GCS 中启动传输，它开始复制文件。它获取文件夹结构，但随后出现错误：

PERMISSION_DENIED: (showing 5 of 104342 failures)
Object: Aperture Library.aplibrary/Database/Albums/L2z6kBLyQPKRGvv8Jp%yig.apalbum
Object: Aperture Library.aplibrary/Database/Albums/LxYeHbbiSgaQzhjBgpbJBg.apalbum
Object: Aperture Library.aplibrary/Database/Albums/TT+4B27jQx+Us1lc3XJszQ.apalbum

PERMISSION_DENIED: (showing 5 of 104342 failures)
Object:
Object:
Object:
Object:
Object:

我已经多次尝试使用 S3 中的组权限设置为管理员，使用内联策略，并将两者结合起来。此外，即使来源不同，我也选中了覆盖 GCS 文件的框。它获取文件夹结构和大量文件，但随后遇到这些对象并停止。

我的下一次尝试是使用 gsutil 找出问题发生的确切位置，但我感觉会出现相同的结果。

有什么建议吗？

谢谢！

==============

更新：

正如预期的那样，gsutil 发出了错误，虽然这次我能够看到它没有立即工作，而不是等待 GCS 传输功能结束。我的输出：

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidObjectState</Code><Message>The operation is not valid for the object's storage class</Message><RequestId>MyReqID</RequestId><HostId>MyHostID</HostId></Error>

Copying s3://benmctee-aperture/Aperture Library.aplibrary/Database/Albums/CtW8oCcESyOINcweX1imtQ.apalbum [Content-Type=application/octet-stream]...
Copying s3://benmctee-aperture/Aperture Library.aplibrary/Database/Albums/DFVs%e5uSwqZ16gNqgycPQ.apalbum [Content-Type=application/octet-stream]...
Copying s3://benmctee-aperture/Aperture Library.aplibrary/Database/Albums/DLYiL8uITfuo6zLqdgYr1w.apalbum [Content-Type=application/octet-stream]...
Copying s3://benmctee-aperture/Aperture Library.aplibrary/Database/Albums/D7UOGRrXS3muB2ilQ80Fmw.apalbum [Content-Type=application/octet-stream]...
Exception in thread Thread-85:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/daisy_chain_wrapper.py", line 197, in PerformDownload
    decryption_tuple=self.decryption_tuple)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 252, in GetObjectMedia
    decryption_tuple=decryption_tuple)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 493, in GetObjectMedia
    generation=generation)
  File "/usr/lib/google-cloud-sdk/platform/gsutil/gslib/boto_translation.py", line 1430, in _TranslateExceptionAndRaise
    raise translated_exception
AccessDeniedException: AccessDeniedException: 403 InvalidObjectState
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidObjectState</Code><Message>The operation is not valid for the object's storage class</Message><RequestId>MyReqID</RequestId><HostId>MyHostID</HostId></Error>

Answer 1

这些对象是否存储在 Amazon Glacier 中？ "InvalidObjectState" 是如果您尝试从 Amazon Glacier 读取对象但尚未 "restored" 对象，S3 会收到的错误。

如果是这种情况，请注意，从 Amazon Glacier 恢复对象可能需要数小时，而且费用可能非常昂贵，具体取决于对象的数量和许多其他因素。

PERMISSION_DENIED 从 S3 转移到 GCS

PERMISSION_DENIED while transferring from S3 to GCS

amazon-s3

google-cloud-storage