使用 Kinesis Firehose 将数据写入 S3 时出现问题，所有记录都是 404 消息

Question

我已经设置了一个 Kinesis Firehose 流，它在 AWS 控制台中成功运行s "test"。我写入流的代码如下所示：

return graphqlHTTP(
(request: Object, response: Object, params: Object): Object => {
  const firehoseConfig: Object = {
    region: config.get('awsRegion'),
    credentials = {
      accessKeyId: config.get('awsAccessKey'),
      secretAccessKey: config.get('secretAccessKey'),
    }
  }
  // Init Kinesis Firehose
  const firehose = new AWS.Firehose(firehoseConfig)
  // Prep response object for sending
  const stringifiedResponse = JSON.stringify(response)
  // Send to Kinesis Firehose stream
  const firehoseParams = {
    DeliveryStreamName: 'test-stream',
    Record: {
      Data: Buffer.from(stringifiedResponse),
    },
  }
  firehose.putRecord(firehoseParams, (err: Object, data: Object) => {
    // eslint-disable-next-line no-console
    if (err) console.log('FIREHOSE ERROR: ', err, err.stack)
    // eslint-disable-next-line no-console
    else console.log(data)
  })

记录在命中端点时写入流，但是所有记录如下所示：

 {
  "status": 404,
  "message": "Not Found",
  "header": {
    "x-frame-options": "SAMEORIGIN",
    "strict-transport-security": "max-age=86400",
    "x-download-options": "noopen",
    "x-content-type-options": "nosniff",
    "x-xss-protection": "1; mode=block",
    "vary": "Accept-Encoding, Origin",
    "cache-control": "max-age=60, s-maxage=60",
    "access-control-allow-origin": "<redacted internal site>",
    "access-control-allow-credentials": "true",
    "access-control-expose-headers": "content-length,etag",
    "x-ratelimit-remaining": "996",
    "x-ratelimit-reset": "1571372307",
    "x-ratelimit-limit": "1000"
  }
} {
  "status": 404,
  "message": "Not Found",
  "header": {
    "x-frame-options": "SAMEORIGIN",
    "strict-transport-security": "max-age=86400",
    "x-download-options": "noopen",
    "x-content-type-options": "nosniff",
    "x-xss-protection": "1; mode=block",
    "vary": "Accept-Encoding, Origin",
    "cache-control": "max-age=60, s-maxage=60",
    "access-control-allow-origin": "<redacted internal site>",
    "access-control-allow-credentials": "true",
    "access-control-expose-headers": "content-length,etag",
    "x-ratelimit-remaining": "995",
    "x-ratelimit-reset": "1571372307",
    "x-ratelimit-limit": "1000"
  }
}

我想做的是让这个 Firehose 将 response 发送到 S3，这样我以后就可以运行使用 Athena 查询它。

Answer 1

事实证明，根据 docs，来自 Kinesis 的 404 指示格式错误的查询。在此上下文中，这意味着 Record 中提供的 Data 无效 JSON。这里Koa提供的response对象/koa-graphql是一个循环引用的大对象，其中某处是AWS认为无效的JSON或JSON无效。

使用 Kinesis Firehose 将数据写入 S3 时出现问题，所有记录都是 404 消息

Problem writing data to S3 with Kinesis Firehose, all records are 404 messages

amazon-web-services

amazon-kinesis