Golang 从 S3 读取 JSON 到内存中的结构

Question

我在 S3 中有一个 JSON 文件，它采用以下结构的格式：

type StockInfo []struct {
    Ticker         string `json:"ticker"`
    BoughtPrice    string `json:"boughtPrice"`
    NumberOfShares string `json:"numberOfShares"`
}

我想将它读入 S3 的结构值。在 python 中，代码看起来像这样：

import boto3
import json

s3 = boto3.client('s3', 'us-east-1')
obj = s3.get_object(Bucket=os.environ["BucketName"], Key=os.environ["Key"])
fileContents = obj['Body'].read().decode('utf-8')
json_content = json.loads(fileContents)

然而，我有点卡在如何在 Go 中实现这一点上。我已经走到这一步了：

package main

import (
    "archive/tar"
    "bytes"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
    "github.com/joho/godotenv"
)

type StockInfo []struct {
    Ticker         string `json:"ticker"`
    BoughtPrice    string `json:"boughtPrice"`
    NumberOfShares string `json:"numberOfShares"`
}

func init() {
    // loads values from .env into the system
    if err := godotenv.Load(); err != nil {
        log.Print("No .env file found")
    }
    return
}

func main() {
    // Store the PATH environment variable in a variable
    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("us-east-1")},
    )
    if err != nil {
        panic(err)
    }

    s3Client := s3.New(sess)
    bucket := "ian-test-bucket-go-python"
    key := "StockInfo.json"

    requestInput := &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }
    result, err := s3Client.GetObject(requestInput)
        if err != nil {
             fmt.Println(err)
        }
    fmt.Println(result)

returns 对我来说 body/object 缓冲区，但我不确定如何将其读入字符串以便将其编组到我的结构中。我在类似的问题中找到了这段代码：

    requestInput := &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }

    buf := new(aws.WriteAtBuffer)
    numBytes, _ := *s3manager.Downloader.Download(buf, requestInput)
    tr := tar.NewReader(bytes.NewReader(buf.Bytes()))

但我收到以下错误：

not enough arguments in call to method expression s3manager.Downloader.Download
    have (*aws.WriteAtBuffer, *s3.GetObjectInput)
    want (s3manager.Downloader, io.WriterAt, *s3.GetObjectInput, ...func(*s3manager.Downloader))

multiple-value s3manager.Downloader.Download() in single-value context

任何人都可以指出我正确的方向吗？与 python.

相比，这样做的难度似乎有点令人沮丧

Answer 1

我可以用下面的代码做到这一点：

    requestInput := &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }

    result, err := s3Client.GetObject(requestInput)
    if err != nil {
        fmt.Println(err)
    }
    defer result.Body.Close()
    body1, err := ioutil.ReadAll(result.Body)
    if err != nil {
        fmt.Println(err)
    }
    bodyString1 := fmt.Sprintf("%s", body1)

    var s3data StockInfo
    decoder := json.NewDecoder(strings.NewReader(bodyString1))
    err = decoder.Decode(&s3data)
    if err != nil {
        fmt.Println("twas an error")
    }

    fmt.Println(s3data)

Answer 2

使用 json.Unmarshal

的替代解决方案

besed on aws-sdk-go-v2


...
params := &s3.GetObjectInput{
    Bucket: aws.String(s3Record.S3.Bucket.Name),
    Key:    aws.String(s3Record.S3.Object.Key),
}
result, _ := client.GetObject(context.TODO(), params)
if err != nil {
    panic(err)
}
defer result.Body.Close()

// capture all bytes from upload
b, err := ioutil.ReadAll(result.Body)
if err != nil {
    panic(err)
}

var temp StockInfo
if err = json.Unmarshal(b, &temp); err != nil {
    panic(err)
}
ftm.Println("res: ",b)

Golang 从 S3 读取 JSON 到内存中的结构

Golang Read JSON from S3 into struct in memory

json

amazon-s3

go

unmarshalling

amazon-web-services