如何使用 boto3 从名称中带有句点 (.) 的存储桶访问密钥?
How to access keys from buckets with periods (.) in their names using boto3?
上下文
我正在尝试获取我所有存储桶的加密状态以用于安全报告。但是,由于加密是基于密钥级别的,因此我想遍历所有密钥并获得一般加密状态。例如,"yes" 是所有密钥都被加密,"no" 如果 none 被加密,而 "partially" 是一些被加密。
我必须使用 boto3,因为 boto 存在一个已知问题,即每个密钥的加密状态总是 returns None。 See here.
问题
我正在尝试使用 boto3 遍历我的每个存储桶中的所有键。以下代码工作正常,直到遇到名称包含句点的存储桶,例如 "my.test.bucket".
from boto3.session import Session
session = Session(aws_access_key_id=<ACCESS_KEY>,
aws_secret_access_key=<SECRET_KEY>,
aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')
for bucket in s3_resource.buckets.all():
for obj in bucket.objects.all():
key = s3_resource.Object(bucket.name, obj.key)
# Do some stuff with the key...
当它命中名称中带有句点的存储桶时,它会在调用 bucket.objects.all()
时抛出此异常,告诉我将所有请求发送到特定端点。这个端点可以在抛出的异常对象中找到。
for obj in bucket.objects.all():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 82, in __iter__
for page in self.pages():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 165, in pages
for page in pages:
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 85, in __iter__
response = self._make_request(current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 157, in _make_request
return self._method(**current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 395, in _make_api_call
raise ClientError(parsed_response, operation_name)botocore.exceptions.ClientError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
我尝试过的东西
- 将 endpoint_url 参数设置为异常响应中指定的存储桶端点,如
s3_resource = session.resource('s3', endpoint_url='my.test.bucket.s3.amazonaws.com')
- 指定存储桶所在的区域,如
s3_resource = session.resource('s3', region_name='eu-west-1')
我认为这个问题类似于 boto 中的 this Whosebug question,它通过在 s3Connection 构造函数中设置 calling_format 参数来解决问题。不幸的是,我不能使用 boto(见上文)。
更新
这是最终对我有用的东西。这不是最优雅的方法,但它有效 =).
from boto3.session import Session
session = Session(aws_access_key_id=<ACCESS_KEY>,
aws_secret_access_key=<SECRET_KEY>,
aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')
# First get all the bucket names
bucket_names = [bucket.name for bucket in s3_resource.buckets.all()]
for bucket_name in bucket_names:
# Check each name for a "." and use a different resource if needed
if "." in bucket_name:
region = session.client('s3').get_bucket_location(Bucket=bucket_name)['LocationConstraint']
resource = session.resource('s3', region_name=region)
else:
resource = s3_resource
bucket = resource.Bucket(bucket_name)
# Continue as usual using this resource
for obj in bucket.objects.all():
key = resource.Object(bucket.name, obj.key)
# Do some stuff with the key...
关于这个有几个 github issues。这与桶的区域有关。确保您的 S3 资源与您创建的存储桶位于同一区域。
FWIW 您可以像这样以编程方式确定区域:
s3.meta.client.get_bucket_location(Bucket='boto3.region')
只是概括了 Ben 提供的出色答案。
import boto3
knownBucket = 'some.topLevel.BucketPath.withPeriods'
s3 = boto3.resource('s3')
#get region
region = s3.meta.client.get_bucket_location(Bucket=knownBucket)['LocationConstraint']
#set region in resource
s3 = boto3.resource('s3',region_name=region)
上下文
我正在尝试获取我所有存储桶的加密状态以用于安全报告。但是,由于加密是基于密钥级别的,因此我想遍历所有密钥并获得一般加密状态。例如,"yes" 是所有密钥都被加密,"no" 如果 none 被加密,而 "partially" 是一些被加密。
我必须使用 boto3,因为 boto 存在一个已知问题,即每个密钥的加密状态总是 returns None。 See here.
问题
我正在尝试使用 boto3 遍历我的每个存储桶中的所有键。以下代码工作正常,直到遇到名称包含句点的存储桶,例如 "my.test.bucket".
from boto3.session import Session
session = Session(aws_access_key_id=<ACCESS_KEY>,
aws_secret_access_key=<SECRET_KEY>,
aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')
for bucket in s3_resource.buckets.all():
for obj in bucket.objects.all():
key = s3_resource.Object(bucket.name, obj.key)
# Do some stuff with the key...
当它命中名称中带有句点的存储桶时,它会在调用 bucket.objects.all()
时抛出此异常,告诉我将所有请求发送到特定端点。这个端点可以在抛出的异常对象中找到。
for obj in bucket.objects.all():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 82, in __iter__
for page in self.pages():
File "/usr/local/lib/python2.7/site-packages/boto3/resources/collection.py", line 165, in pages
for page in pages:
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 85, in __iter__
response = self._make_request(current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/paginate.py", line 157, in _make_request
return self._method(**current_kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 310, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 395, in _make_api_call
raise ClientError(parsed_response, operation_name)botocore.exceptions.ClientError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
我尝试过的东西
- 将 endpoint_url 参数设置为异常响应中指定的存储桶端点,如
s3_resource = session.resource('s3', endpoint_url='my.test.bucket.s3.amazonaws.com')
- 指定存储桶所在的区域,如
s3_resource = session.resource('s3', region_name='eu-west-1')
我认为这个问题类似于 boto 中的 this Whosebug question,它通过在 s3Connection 构造函数中设置 calling_format 参数来解决问题。不幸的是,我不能使用 boto(见上文)。
更新
这是最终对我有用的东西。这不是最优雅的方法,但它有效 =).
from boto3.session import Session
session = Session(aws_access_key_id=<ACCESS_KEY>,
aws_secret_access_key=<SECRET_KEY>,
aws_session_token=<TOKEN>)
s3_resource = session.resource('s3')
# First get all the bucket names
bucket_names = [bucket.name for bucket in s3_resource.buckets.all()]
for bucket_name in bucket_names:
# Check each name for a "." and use a different resource if needed
if "." in bucket_name:
region = session.client('s3').get_bucket_location(Bucket=bucket_name)['LocationConstraint']
resource = session.resource('s3', region_name=region)
else:
resource = s3_resource
bucket = resource.Bucket(bucket_name)
# Continue as usual using this resource
for obj in bucket.objects.all():
key = resource.Object(bucket.name, obj.key)
# Do some stuff with the key...
关于这个有几个 github issues。这与桶的区域有关。确保您的 S3 资源与您创建的存储桶位于同一区域。
FWIW 您可以像这样以编程方式确定区域:
s3.meta.client.get_bucket_location(Bucket='boto3.region')
只是概括了 Ben 提供的出色答案。
import boto3
knownBucket = 'some.topLevel.BucketPath.withPeriods'
s3 = boto3.resource('s3')
#get region
region = s3.meta.client.get_bucket_location(Bucket=knownBucket)['LocationConstraint']
#set region in resource
s3 = boto3.resource('s3',region_name=region)