Python generate_blob_sas 没有创建正确的 SAS 令牌（azcopy 中的复制操作失败）
Python generate_blob_sas doesn't create right SAS token (failed on copy operation in azcopy)
这是我在这么酷的资源上的第一个问题。
目标描述：我需要从一个 Azure 存储帐户复制到另一个。 ... 不仅如此，这部分现在对我不起作用。
问题： 如果使用 generate_blob_sas 生成令牌，您将在 azcopy 操作中遇到：
INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.13.1-0.20210914164749-2d6cd3e07548/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) =====
Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
如果在 Azure 门户上插入手动创建的令牌 - 工作正常。
为什么我在 Python 中使用 azcopy，因为我有这样的结构：
.
|
|└── Folder1
| ├── file1.json
| ├── file2.json
| ├── file3.json
| ├── file4.json
| ├── file5.son
| └── Folder2
| ├── fileswithsomenames1.html
| ├── fileswithsomenames2.html
| ├── fileswithsomenames3.html
| ├── fileswithsomenames4.html
| ├── fileswithsomenames5.html
| ├── fileswithsomenames6.html
| ├── fileswithsomenames7.html
| ├── fileswithsomenames8.html
| ├── fileswithsomenames9.html
| └── fileswithsomenames10.html
└──receipt.json (this file will be eachtime but with different content)
每种情况下的文件数量会有所不同。我需要继续处理来自目标 blob 的文件。
我没有找到可以理解的描述如何从 store1/conteiner1/{var.name}-blob 复制到 var.name-store2/conteainer1/ 包括许多文件夹在内的所有内容。几乎所有 1 个 blob 文件的描述和手册。
我的代码：
code to authorise and get storage account name and string
...
customer_name = 'abc-qa'
...
container_name_source = "container1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"
container_name_target = customer_name+"-bl"
blob_name_target = "/"
# Function to generate sas for source blob
def get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source):
sas_blob_source = generate_blob_sas(account_name=account_name_source,
container_name=container_name_source,
blob_name=blob_name_source,
account_key=account_key_1_source,
permission=BlobSasPermissions(read=True),
expiry=datetime.utcnow() + timedelta(hours=1))
return sas_blob_source
sas_blob_source = get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source)
url_blob_source = 'https://'+account_name_source+'.blob.core.windows.net/'+container_name_source+'/'+blob_name_source+'?'+sas_blob_source
# Function to generate sas for target blob
def get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target):
sas_blob_target = generate_blob_sas(account_name=account_name_target,
container_name=container_name_target,
blob_name=blob_name_target,
account_key=account_key_1_target,
permission=BlobSasPermissions(add=True, create=True, write=True, tag=False, delete_previous_version=True),
expiry=datetime.utcnow() + timedelta(hours=1))
return sas_blob_target
sas_blob_target = get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target)
print(sas_blob_target)
url_blob_target = 'https://'+account_name_target+'.blob.core.windows.net/'+container_name_target+'/?'+sas_blob_target
#copy seed-data folder from account_name_source/container1/customer_name+"-blobfolder1/blobfolder2" to account_name_target/customer_name+"-bl"
cmd = f"azcopy copy '{url_blob_source}' '{url_blob_target}' --recursive=True"
print(str(cmd))
results = subprocess.run(
str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)
手动生成的结果
'sp=rl&st=2022-04-14T14:29:48Z&se=2022-04-14T22:29:48Z&spr=https&sv=2020-08-04&sr=c&sig={token}'
Rusult if gen over generate_blob_sas
'se=2022-04-14T17%3A18%3A48Z&sp=r&sv=2020-10-02&sr=b&sig={token}'
您的代码以编程方式生成的 SAS 令牌失败的原因是因为它缺少 List 权限。如果您查看正在运行的 SAS 令牌，它同时具有 Read 和 List 权限 (sp=rl)，而 non-working SAS 令牌仅具有 Read 权限（sp=r).
另一个问题是，在您的代码中，您正在为 blob (sr=b) 生成 SAS 令牌，而工作的 SAS 令牌是为容器 (sr=c) 生成的。这适用于源和目标 SAS 令牌。
解决这个问题：
- 创建具有
Read 和 List 权限的 SAS 令牌。
- 为容器而不是 blob 创建 SAS 令牌。为此，您需要使用
generate_container_sas.
一旦你做了这两件事，你应该不会得到错误。
这是我现在可以使用的工作代码 generate_container_sas 来自访问策略：
# Here code where I get storage account name and key. And Authentication by SPN which in KV.
...
customer_name = 'abc-qa'
container_name_source = "conteiner1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"
container_name_target = customer_name+"-bl"
blob_name_target = "blobfolder2"
from datetime import datetime, timedelta
connection_string_source = 'DefaultEndpointsProtocol=https;AccountName='+account_name_source+';AccountKey='+blob_key_1_source+';EndpointSuffix=core.windows.net'
connection_string_target = 'DefaultEndpointsProtocol=https;AccountName='+account_name_target+';AccountKey='+blob_key_1_target+';EndpointSuffix=core.windows.net'
# Instantiate a BlobServiceClient using a connection string
from azure.storage.blob import BlobServiceClient
blob_service_client_source = BlobServiceClient.from_connection_string(connection_string_source)
blob_service_client_target = BlobServiceClient.from_connection_string(connection_string_target)
# Instantiate a ContainerClient
container_client_source = blob_service_client_source.get_container_client(container_name_source)
container_client_target = blob_service_client_target.get_container_client(container_name_target)
# [START set_container_access_policy]
# Create access policy Soure
from azure.storage.blob import AccessPolicy, ContainerSasPermissions
access_policy_source = AccessPolicy(permission=ContainerSasPermissions(read=True, list=True),
expiry=datetime.utcnow() + timedelta(hours=1),
start=datetime.utcnow() - timedelta(minutes=1))
identifiers_source = {'rl': access_policy_source}
# Create access policy Target
from azure.storage.blob import AccessPolicy, ContainerSasPermissions
access_policy_target = AccessPolicy(permission=ContainerSasPermissions(write=True),
expiry=datetime.utcnow() + timedelta(hours=1),
start=datetime.utcnow() - timedelta(minutes=1))
identifiers_target = {'w': access_policy_target}
# Set the access policy on the container
container_client_source.set_container_access_policy(signed_identifiers=identifiers_source)
container_client_target.set_container_access_policy(signed_identifiers=identifiers_target)
# [END set_container_access_policy]
# [START get_container_access_policy]
policy_source = container_client_source.get_container_access_policy()
policy_source = container_client_target.get_container_access_policy()
# [END get_container_access_policy]
# [START generate_sas_token]
# Use access policy to generate a sas token Source
from azure.storage.blob import generate_container_sas
sas_token_source = generate_container_sas(
container_client_source.account_name,
container_client_source.container_name,
account_key=container_client_source.credential.account_key,
policy_id='rl'
)
print(sas_token_source)
# [END generate_sas_token]
# Use access policy to generate a sas token Source
from azure.storage.blob import generate_container_sas
sas_token_target = generate_container_sas(
container_client_target.account_name,
container_client_target.container_name,
account_key=container_client_target.credential.account_key,
policy_id='w'
)
print(sas_token_target)
# [END generate_sas_token]
#copy seed-data folder from stcorpmainstorage/seed-data/$(name of env)-seed-data/seed-data to $(name of env)store
cmd = f"azcopy copy 'https://{account_name_source}.blob.core.windows.net/{container_name_source}/{blob_name_source}?{sas_token_source}' 'https://{account_name_target}.blob.core.windows.net/{container_name_target}/?{sas_token_target}' --recursive=True"
results = subprocess.run(
str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)
...
# Here code to work with target blob and next steps.
希望对大家有用。
这是我在这么酷的资源上的第一个问题。
目标描述：我需要从一个 Azure 存储帐户复制到另一个。 ... 不仅如此，这部分现在对我不起作用。
问题： 如果使用 generate_blob_sas 生成令牌，您将在 azcopy 操作中遇到：
INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go@v0.13.1-0.20210914164749-2d6cd3e07548/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=AuthenticationFailed) ===== Description=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
如果在 Azure 门户上插入手动创建的令牌 - 工作正常。
为什么我在 Python 中使用 azcopy，因为我有这样的结构：
.
|
|└── Folder1
| ├── file1.json
| ├── file2.json
| ├── file3.json
| ├── file4.json
| ├── file5.son
| └── Folder2
| ├── fileswithsomenames1.html
| ├── fileswithsomenames2.html
| ├── fileswithsomenames3.html
| ├── fileswithsomenames4.html
| ├── fileswithsomenames5.html
| ├── fileswithsomenames6.html
| ├── fileswithsomenames7.html
| ├── fileswithsomenames8.html
| ├── fileswithsomenames9.html
| └── fileswithsomenames10.html
└──receipt.json (this file will be eachtime but with different content)
每种情况下的文件数量会有所不同。我需要继续处理来自目标 blob 的文件。 我没有找到可以理解的描述如何从 store1/conteiner1/{var.name}-blob 复制到 var.name-store2/conteainer1/ 包括许多文件夹在内的所有内容。几乎所有 1 个 blob 文件的描述和手册。
我的代码：
code to authorise and get storage account name and string
...
customer_name = 'abc-qa'
...
container_name_source = "container1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"
container_name_target = customer_name+"-bl"
blob_name_target = "/"
# Function to generate sas for source blob
def get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source):
sas_blob_source = generate_blob_sas(account_name=account_name_source,
container_name=container_name_source,
blob_name=blob_name_source,
account_key=account_key_1_source,
permission=BlobSasPermissions(read=True),
expiry=datetime.utcnow() + timedelta(hours=1))
return sas_blob_source
sas_blob_source = get_blob_sas_source(account_name_source,account_key_1_source, container_name_source, blob_name_source)
url_blob_source = 'https://'+account_name_source+'.blob.core.windows.net/'+container_name_source+'/'+blob_name_source+'?'+sas_blob_source
# Function to generate sas for target blob
def get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target):
sas_blob_target = generate_blob_sas(account_name=account_name_target,
container_name=container_name_target,
blob_name=blob_name_target,
account_key=account_key_1_target,
permission=BlobSasPermissions(add=True, create=True, write=True, tag=False, delete_previous_version=True),
expiry=datetime.utcnow() + timedelta(hours=1))
return sas_blob_target
sas_blob_target = get_blob_sas_target(account_name_target,account_key_1_target, container_name_target, blob_name_target)
print(sas_blob_target)
url_blob_target = 'https://'+account_name_target+'.blob.core.windows.net/'+container_name_target+'/?'+sas_blob_target
#copy seed-data folder from account_name_source/container1/customer_name+"-blobfolder1/blobfolder2" to account_name_target/customer_name+"-bl"
cmd = f"azcopy copy '{url_blob_source}' '{url_blob_target}' --recursive=True"
print(str(cmd))
results = subprocess.run(
str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)
手动生成的结果 'sp=rl&st=2022-04-14T14:29:48Z&se=2022-04-14T22:29:48Z&spr=https&sv=2020-08-04&sr=c&sig={token}'
Rusult if gen over generate_blob_sas
'se=2022-04-14T17%3A18%3A48Z&sp=r&sv=2020-10-02&sr=b&sig={token}'
您的代码以编程方式生成的 SAS 令牌失败的原因是因为它缺少 List 权限。如果您查看正在运行的 SAS 令牌，它同时具有 Read 和 List 权限 (sp=rl)，而 non-working SAS 令牌仅具有 Read 权限（sp=r).
另一个问题是，在您的代码中，您正在为 blob (sr=b) 生成 SAS 令牌，而工作的 SAS 令牌是为容器 (sr=c) 生成的。这适用于源和目标 SAS 令牌。
解决这个问题：
- 创建具有
Read和List权限的 SAS 令牌。 - 为容器而不是 blob 创建 SAS 令牌。为此，您需要使用
generate_container_sas.
一旦你做了这两件事，你应该不会得到错误。
这是我现在可以使用的工作代码 generate_container_sas 来自访问策略：
# Here code where I get storage account name and key. And Authentication by SPN which in KV.
...
customer_name = 'abc-qa'
container_name_source = "conteiner1"
blob_name_source = customer_name+"-blobfolder1/blobfolder2"
container_name_target = customer_name+"-bl"
blob_name_target = "blobfolder2"
from datetime import datetime, timedelta
connection_string_source = 'DefaultEndpointsProtocol=https;AccountName='+account_name_source+';AccountKey='+blob_key_1_source+';EndpointSuffix=core.windows.net'
connection_string_target = 'DefaultEndpointsProtocol=https;AccountName='+account_name_target+';AccountKey='+blob_key_1_target+';EndpointSuffix=core.windows.net'
# Instantiate a BlobServiceClient using a connection string
from azure.storage.blob import BlobServiceClient
blob_service_client_source = BlobServiceClient.from_connection_string(connection_string_source)
blob_service_client_target = BlobServiceClient.from_connection_string(connection_string_target)
# Instantiate a ContainerClient
container_client_source = blob_service_client_source.get_container_client(container_name_source)
container_client_target = blob_service_client_target.get_container_client(container_name_target)
# [START set_container_access_policy]
# Create access policy Soure
from azure.storage.blob import AccessPolicy, ContainerSasPermissions
access_policy_source = AccessPolicy(permission=ContainerSasPermissions(read=True, list=True),
expiry=datetime.utcnow() + timedelta(hours=1),
start=datetime.utcnow() - timedelta(minutes=1))
identifiers_source = {'rl': access_policy_source}
# Create access policy Target
from azure.storage.blob import AccessPolicy, ContainerSasPermissions
access_policy_target = AccessPolicy(permission=ContainerSasPermissions(write=True),
expiry=datetime.utcnow() + timedelta(hours=1),
start=datetime.utcnow() - timedelta(minutes=1))
identifiers_target = {'w': access_policy_target}
# Set the access policy on the container
container_client_source.set_container_access_policy(signed_identifiers=identifiers_source)
container_client_target.set_container_access_policy(signed_identifiers=identifiers_target)
# [END set_container_access_policy]
# [START get_container_access_policy]
policy_source = container_client_source.get_container_access_policy()
policy_source = container_client_target.get_container_access_policy()
# [END get_container_access_policy]
# [START generate_sas_token]
# Use access policy to generate a sas token Source
from azure.storage.blob import generate_container_sas
sas_token_source = generate_container_sas(
container_client_source.account_name,
container_client_source.container_name,
account_key=container_client_source.credential.account_key,
policy_id='rl'
)
print(sas_token_source)
# [END generate_sas_token]
# Use access policy to generate a sas token Source
from azure.storage.blob import generate_container_sas
sas_token_target = generate_container_sas(
container_client_target.account_name,
container_client_target.container_name,
account_key=container_client_target.credential.account_key,
policy_id='w'
)
print(sas_token_target)
# [END generate_sas_token]
#copy seed-data folder from stcorpmainstorage/seed-data/$(name of env)-seed-data/seed-data to $(name of env)store
cmd = f"azcopy copy 'https://{account_name_source}.blob.core.windows.net/{container_name_source}/{blob_name_source}?{sas_token_source}' 'https://{account_name_target}.blob.core.windows.net/{container_name_target}/?{sas_token_target}' --recursive=True"
results = subprocess.run(
str(cmd), shell=True, universal_newlines=True, check=True)
print(results.stdout)
...
# Here code to work with target blob and next steps.
希望对大家有用。