无法使用 boto3 将多个对象上传到 google 存储

Question

在下面粘贴我的代码片段：

args.tenant = "test-conversenow"
    try:
        print("Uploading models & sdk to spaces bucket...")
        gcs.meta.client.upload_file(core_dep, args.tenant, 'core/' + model_version)
        gcs.meta.client.upload_file(nlu_dep, args.tenant, 'nlu/' + model_version)
        gcs.meta.client.upload_file(sdk_dep, args.tenant, 'sdk/' + model_version)
        gcs.meta.client.upload_file(data_dep, args.tenant, 'data/' + model_version)
        print("Uploading done to bucket")
    except Exception as err:
        print("Error uploading to bucket: ", err)

我得到的错误是

Error uploading to bucket: Failed to upload deployment/nlu/0.0.1.tar.gz to test-conversenow/nlu/0.0.1.tar.gz: An error occurred (InvalidArgument) when calling the CreateMultipartUpload operation: Invalid argument.

第一个对象已创建，然后我的程序在尝试上传第二个对象时失败 nlu/0.0.1.tar.gz

Answer 1

我在最近的示例工作中使用了这个，下面是代码；

>>> import boto3
>>> boto3.__version__
'1.9.203'



import boto3

s3 = boto3.resource('s3') # s3 object to upload.
local_file_path = local_folder_path + "\"+ file_name_to_upload # local_file_path = D:\Boto3Sample\sample.txt
s3.meta.client.upload_file(local_file_path, "s3BucketName", file_name_to_upload) # upload_file("D:\Boto3Sample\sample.txt", "s3BucketName", "sample.txt")

从您的错误消息来看，Key 的值（根据文档，"The name of the key to upload to" 第三个参数）在上传到存储时出现了一些问题。

Answer 2

我使用 s3transfer 修复了它。下面是代码片段。

import boto3
from botocore.client import Config
from s3transfer import TransferConfig, S3Transfer


myconfig = TransferConfig(
        multipart_threshold=9999999999999999,
        max_concurrency=10,
        num_download_attempts=10,
    )

client = boto3.client(service_name='s3',
                      region_name=REGION_NAME,
                      endpoint_url=ENDPOINT_URL,
                      aws_access_key_id=CLIENT_ID,
                      aws_secret_access_key=CLIENT_SECRET,
                      config=Config(signature_version='s3v4'))
transfer = S3Transfer(client, myconfig)

try:
    print("Uploading models & sdk to spaces bucket...")
    transfer.upload_file(core_dep, args.tenant, 'core/' + model_version)
    transfer.upload_file(nlu_dep, args.tenant, 'nlu/' + model_version)
    transfer.upload_file(data_dep, args.tenant, 'data/' + model_version)
    transfer.upload_file(sdk_dep, args.tenant, 'sdk/' + model_version)
    print("Uploading done to bucket")
except Exception as err:
    print("Error uploading to bucket: ", err)

无法使用 boto3 将多个对象上传到 google 存储

Fail to upload multiple objects with boto3 to google storage

python

boto

google-cloud-storage

boto3