将文件从 lambda 推送到 s3

Question

我已经使用运动流触发了 lambda，并在其中查找操作被阻止的记录，并将数据附加到输出文件。

如何将该文件推送到 s3？我在下面写了但不确定。

新代码导入json

import urllib.parse
import boto3

print('Loading function')


s3 = boto3.client('s3')

def lambda_handler(event, context):

    #1 - Get the bucket name
    bucket = event['Records'][0]['s3']['bucket']['name']

    #2 - Get the file/key name
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')


    #3 - Fetch the file from S3
    response = s3.get_object(Bucket=bucket, Key=key)

    #4 - Deserialize the file's content
    text = response["Body"].read().decode()
    e  = text.split("\n")

    Output=[]
    #5 - Print the content
    print(text)

    #6 - Parse and print the Action

    for each in e:
        loaded_data = json.loads(e)
        if loaded_data["action"] == "ALLOW":
            print("dropped")
        else :    
            Output.append(loaded_data)
    s3.put_object(Body='json.dumps(output)',Bucket='blocketreques',Key='Filtered.txt')
    print('Put Complete')

import json
import urllib.parse
import boto3

print('Loading function')


s3 = boto3.client('s3')

def lambda_handler(event, context):

    #1 - Get the bucket name
    bucket = event['Records'][0]['s3']['bucket']['name']

    #2 - Get the file/key name
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')


    #3 - Fetch the file from S3
    response = s3.get_object(Bucket=bucket, Key=key)

    #4 - Deserialize the file's content
    text = response["Body"].read().decode()
    e  = text.split("\n")

    Output=[]
    #5 - Print the content
    print(text)

    #6 - Parse and print the Action

    for each in e:
        loaded_data = json.loads(e)
        if loaded_data["action"] == "ALLOW":
            print("dropped")
        else :    
            Output.append(loaded_data)
    s3.put_object(Body='json.dumps(output)',Bucket='blocketreques',Key='Filtered.txt')
    print('Put Complete')

Answer 1

代码使用 s3.upload_file()，它 从磁盘上传文件 。

如果你想从内存中上传内容（例如output数组），你可以使用：

s3.put_object(Body=json.dumps(output), Bucket=..., Key=...)

Answer 2

我建议只使用 smart-open

它的行为就像任何其他打开的文件一样，但如果你给它 s3 路径，它会将它保存到 s3。如果您尝试保存更大的文件，它还会处理其他情况，例如分段上传。

PS：在本地测试时，您只需更改输出路径以指向本地机器并加快测试速度

将文件从 lambda 推送到 s3

To push the file from lambda to s3

python

amazon-web-services

aws-lambda

amazon-kinesis-firehose