使用 Amazon S3 智能分层,您如何知道对象位于哪个层?
With Amazon S3 Intelligent Tiering, how do you know which tier an object is in?
https://docs.aws.amazon.com/AmazonS3/latest/userguide/archived-objects.html 说:
When you archive Amazon S3 objects to the S3 Glacier or S3 Glacier
Deep Archive storage class, or when objects are archived to the S3
Intelligent-Tiering Archive Access or Deep Archive Access tiers, the
objects are not accessible in real time.
我明白这一点,但是您如何判断一个对象是在存档访问层还是深度存档访问层?我浏览了大部分 AWS S3 文档,但似乎找不到此信息。我正在为 S3 编写一个 UI 并希望根据对象所在的层显示“下载”或“恢复”按钮。我刚刚开始开发这个所以我没有这两个层中的任何文件都没有,我等不及 90 天才能将任何文件移到那里。
当您执行 list-objects
请求时,您会获得每个项目的存储 class,返回为 INTELLIGENT_TIERING
,但它不会告诉您它是哪一层.
Testing S3 Intelligent Tiering 上的评论说您可以通过将存储 class 更改为 Glacier 来伪造它,但我怀疑这是一个准确的测试,因为 Glacier 和 Intelligent Tiering 是两种不同的存储class是的,智能分层对象突然改变它们的存储是没有意义的 class。
有没有人有任何见解?提前致谢。
来自亚马逊的更新:(tl;博士,这是不可能的)
Unfortunately we cannot identify the access tier for an object stored in Intelligent Tiering class the only way to determine the access tier of an object in by identifying the access pattern of an object. The access pattern can be identified either by analyzing AccessLogs or a user can review access pattern using S3 Storage Lens.
Based on your requirement, the best option for you would be to keep Intelligent-Tiering Archive configuration disabled. With this config disabled the objects in Intelligent-Tiering class will not move in "Archive Access tier" and "Deep Archive Access tier". With this configuration you can create a logic in your application that if the object is in class "Intelligent-Tiering" then the download button will be enabled and user will be able to download the object.
Now for moving objects to "Glacier" and "Glacier Deep Archive" you can configure a life cycle rule. Lets say you can configure a life cycle rule to transition objects in "Intelligent-Tiering" class to Glacier after 90 days and to "Glacier Deep Archive" after 180 days.
This way using ListObjects API call you can identify the correct storage class of the objects and you can program a logic where the "Restore" button will only be enabled if the storage class is "Glacier" or "Glacier Deep Archive". Appropriately a message will be displayed to users to download the object after 3-5 hours.
你可以试一试。尝试调用恢复并捕获异常。然后在你的捕获中你可以做一个 GetObject。
在与 AWS 支持人员反复沟通后,我想出了一个可行的解决方案。对于存储桶中的每个项目,我可以执行 GetObject 请求并请求一个小字节范围(如 10 个字节),因此这几乎等同于执行 HeadObject 请求。如果文件位于 Archive 或 Deep Archive 访问层中,则 AWS 将 return 例外:
HTTP/1.1 403 Forbidden
x-amz-request-id: 43XTDH6Q7E9F2N5J
x-amz-id-2: 3Fn/qzcZwOxaCg1LLDsczSXF7stl7sbOwwDR1ajHIc+guZgxAfFHm1WfRoXWEsFLWPJzKt9Ynaw=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 23 Jun 2021 16:36:24 GMT
Server: AmazonS3
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidObjectState</Code>
<Message>The operation is not valid for the object's access tier</Message>
<StorageClass>INTELLIGENT_TIERING</StorageClass>
<RequestId>43XTDH6Q7E9F2N5J</RequestId>
<HostId>3Fn/qzcZwOxaCg1LLDsczSXF7stl7sbOwwDR1ajHIc+guZgxAfFHm1WfRoXWEsFLWPJzKt9Ynaw=</HostId>
</Error>
所以根据请求是否成功,我可以决定是否显示该项目的“下载”或“恢复”按钮。
@howcheng 您可以在您的对象上使用 S3 Inventory to get list of all of your objects and their corresponding metadata, including the S3 Intelligent-Tiering access tiers (Frequent, Infrequent, Archive, Deep Archive). Amazon S3 Inventory provides CSV, ORC, or Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix. You can also make a HEAD request 来报告 S3 智能分层存档访问层。
来自官方documentation:
You can also use a HEAD object request to view an object's archive
status. If an object is stored using the S3 Intelligent-Tiering
storage class and is in one of the archive tiers, the HEAD object
response shows the current archive tier. It does this using the
x-amz-archive-status
header.
示例:
HTTP/1.1 200 OK
x-amz-id-2: FSVaTMjrmBp3Izs1NnwBZeu7M19iI8UbxMbi0A8AirHANJBo+hEftBuiESACOMJp
x-amz-request-id: E5CEFCB143EB505A
Date: Fri, 13 Nov 2020 00:28:38 GMT
Last-Modified: Mon, 15 Oct 2012 21:58:07 GMT
ETag: "1accb31fcf202eba0c0f41fa2f09b4d7"
x-amz-storage-class: 'INTELLIGENT_TIERING'
x-amz-archive-status: 'ARCHIVE_ACCESS'
x-amz-restore: 'ongoing-request="true"'
x-amz-restore-request-date: 'Fri, 13 Nov 2020 00:20:00 GMT'
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 300
Server: AmazonS3
https://docs.aws.amazon.com/AmazonS3/latest/userguide/archived-objects.html 说:
When you archive Amazon S3 objects to the S3 Glacier or S3 Glacier Deep Archive storage class, or when objects are archived to the S3 Intelligent-Tiering Archive Access or Deep Archive Access tiers, the objects are not accessible in real time.
我明白这一点,但是您如何判断一个对象是在存档访问层还是深度存档访问层?我浏览了大部分 AWS S3 文档,但似乎找不到此信息。我正在为 S3 编写一个 UI 并希望根据对象所在的层显示“下载”或“恢复”按钮。我刚刚开始开发这个所以我没有这两个层中的任何文件都没有,我等不及 90 天才能将任何文件移到那里。
当您执行 list-objects
请求时,您会获得每个项目的存储 class,返回为 INTELLIGENT_TIERING
,但它不会告诉您它是哪一层.
Testing S3 Intelligent Tiering 上的评论说您可以通过将存储 class 更改为 Glacier 来伪造它,但我怀疑这是一个准确的测试,因为 Glacier 和 Intelligent Tiering 是两种不同的存储class是的,智能分层对象突然改变它们的存储是没有意义的 class。
有没有人有任何见解?提前致谢。
来自亚马逊的更新:(tl;博士,这是不可能的)
Unfortunately we cannot identify the access tier for an object stored in Intelligent Tiering class the only way to determine the access tier of an object in by identifying the access pattern of an object. The access pattern can be identified either by analyzing AccessLogs or a user can review access pattern using S3 Storage Lens.
Based on your requirement, the best option for you would be to keep Intelligent-Tiering Archive configuration disabled. With this config disabled the objects in Intelligent-Tiering class will not move in "Archive Access tier" and "Deep Archive Access tier". With this configuration you can create a logic in your application that if the object is in class "Intelligent-Tiering" then the download button will be enabled and user will be able to download the object.
Now for moving objects to "Glacier" and "Glacier Deep Archive" you can configure a life cycle rule. Lets say you can configure a life cycle rule to transition objects in "Intelligent-Tiering" class to Glacier after 90 days and to "Glacier Deep Archive" after 180 days.
This way using ListObjects API call you can identify the correct storage class of the objects and you can program a logic where the "Restore" button will only be enabled if the storage class is "Glacier" or "Glacier Deep Archive". Appropriately a message will be displayed to users to download the object after 3-5 hours.
你可以试一试。尝试调用恢复并捕获异常。然后在你的捕获中你可以做一个 GetObject。
在与 AWS 支持人员反复沟通后,我想出了一个可行的解决方案。对于存储桶中的每个项目,我可以执行 GetObject 请求并请求一个小字节范围(如 10 个字节),因此这几乎等同于执行 HeadObject 请求。如果文件位于 Archive 或 Deep Archive 访问层中,则 AWS 将 return 例外:
HTTP/1.1 403 Forbidden
x-amz-request-id: 43XTDH6Q7E9F2N5J
x-amz-id-2: 3Fn/qzcZwOxaCg1LLDsczSXF7stl7sbOwwDR1ajHIc+guZgxAfFHm1WfRoXWEsFLWPJzKt9Ynaw=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Wed, 23 Jun 2021 16:36:24 GMT
Server: AmazonS3
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>InvalidObjectState</Code>
<Message>The operation is not valid for the object's access tier</Message>
<StorageClass>INTELLIGENT_TIERING</StorageClass>
<RequestId>43XTDH6Q7E9F2N5J</RequestId>
<HostId>3Fn/qzcZwOxaCg1LLDsczSXF7stl7sbOwwDR1ajHIc+guZgxAfFHm1WfRoXWEsFLWPJzKt9Ynaw=</HostId>
</Error>
所以根据请求是否成功,我可以决定是否显示该项目的“下载”或“恢复”按钮。
@howcheng 您可以在您的对象上使用 S3 Inventory to get list of all of your objects and their corresponding metadata, including the S3 Intelligent-Tiering access tiers (Frequent, Infrequent, Archive, Deep Archive). Amazon S3 Inventory provides CSV, ORC, or Parquet output files that list your objects and their corresponding metadata on a daily or weekly basis for an S3 bucket or a shared prefix. You can also make a HEAD request 来报告 S3 智能分层存档访问层。
来自官方documentation:
You can also use a HEAD object request to view an object's archive status. If an object is stored using the S3 Intelligent-Tiering storage class and is in one of the archive tiers, the HEAD object response shows the current archive tier. It does this using the
x-amz-archive-status
header.
示例:
HTTP/1.1 200 OK
x-amz-id-2: FSVaTMjrmBp3Izs1NnwBZeu7M19iI8UbxMbi0A8AirHANJBo+hEftBuiESACOMJp
x-amz-request-id: E5CEFCB143EB505A
Date: Fri, 13 Nov 2020 00:28:38 GMT
Last-Modified: Mon, 15 Oct 2012 21:58:07 GMT
ETag: "1accb31fcf202eba0c0f41fa2f09b4d7"
x-amz-storage-class: 'INTELLIGENT_TIERING'
x-amz-archive-status: 'ARCHIVE_ACCESS'
x-amz-restore: 'ongoing-request="true"'
x-amz-restore-request-date: 'Fri, 13 Nov 2020 00:20:00 GMT'
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 300
Server: AmazonS3