将 json 转换为 csv，然后使用 python 将相同的文件上传到 s3 存储桶

Question

我有一些 json 然后将其转换为 csv 文件，现在同一个文件应该保存到 s3 而不是我的本地文件夹。

  logs = {
"testing1": "testing1_value", 
"testing2": "testing2_value",
"testing3": {"testing3a": "testing1_value3a"}, 
"testing4": {"testing4a": {"testing4a1": "testing_value4a1"}}
}
file_name = "testing_file.csv"
bucket_name = "testing_bucket"
file_to_save_in_path = "path_in_s3/testing_file.csv"
client = boto3.client("s3")


from fastapi.responses import StreamingResponse
stream = await create_csv_for_download(logs, file_name)
    
response = StreamingResponse(iter([stream]), media_type="text/csv")
response.headers["Content-Disposition"] = f"attachment; filename={file_name}"


client.put_object(Bucket=bucket_name, Key=file_to_save_in_path, Body=response)

client.upload_file(response, bucket_name, file_to_save_in_path)

现在响应像一些东西一样 =>

如何将 s3 中的响应保存在适当的 csv 文件中..

使用client.put_object时出错：如下图

**

Parameter validation failed:
Invalid type for parameter Body, value: <starlette.responses.StreamingResponse object at 0x7fe084e75fd0>, type: <class 'starlette.responses.StreamingResponse'>, valid types: <class 'bytes'>, <class 'bytearray'>, file-like object

**

Answer 1

错误信息相当清楚：

Invalid type for parameter Body, value: <starlette.responses.StreamingResponse object at 0x7fe084e75fd0>, 
  type: <class 'starlette.responses.StreamingResponse'>, 
  valid types: <class 'bytes'>, <class 'bytearray'>, file-like object

它告诉你不能将StreamingResponse object传递给put_object()，它必须是一个字节数组，或者一个文件object。假设您的 create_csv_for_download() 函数 returns 是一个流 object，您应该只从中读取字节，然后将其发送到 put_object()。

此外，您在 StreamingResponse 中设置的 HTTP headers 应该直接传递给 put_object()：

import boto3
import csv
import io

def create_csv_for_download(logs, filename):
    # Just a stub so this is a self-contained example
    ret = io.StringIO()
    cw = csv.writer(ret)
    for key, value in logs.items():
        cw.writerow([key, str(value)])
    return ret

logs = {
    "testing1": "testing1_value", 
    "testing2": "testing2_value",
    "testing3": {"testing3a": "testing1_value3a"}, 
    "testing4": {"testing4a": {"testing4a1": "testing_value4a1"}}
}

file_name = "testing_file.csv"
bucket_name = "testing_bucket"
file_to_save_in_path = "path_in_s3/testing_file.csv"
client = boto3.client("s3")

stream = create_csv_for_download(logs, file_name)

# Ready out the body from the stream returned
body = stream.read()
if isinstance(body, str):
    # If this stream returns a string, encode it to a byte array
    body = body.encode("utf-8")

client.put_object(
    Bucket=bucket_name, 
    Key=file_to_save_in_path, 
    Body=body, 
    ContentDisposition=f"attachment; filename={file_name}", 
    ContentType="test/csv",
    # Uncomment the following line if you want the link to be 
    # publicly  downloadable from S3 without credentials:
    # ACL="public-read",
)

将 json 转换为 csv，然后使用 python 将相同的文件上传到 s3 存储桶

convert json to csv and then upload same file to s3 bucket using python

python-3.x

amazon-s3

boto3