Kinesis Firehose Putrecordbatch 示例

Question

我正在寻找使用 putrecordbatch 将多条记录发送到运动流的示例。

我目前正在通过以下方式使用 putrecord 命令将记录发送到运动流。

aws firehose put-record --delivery-stream-name csvtoredshiftstreamingjson --record='Data="{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"'

以类似的方式请求帮助来编写 putrecordbatch。

感谢和问候， 斯里维涅什 KN

Answer 1

来自aws firehose put-record-batch help：

aws firehose put-record-batch \
        --delivery-stream-name csvtoredshiftstreamingjson \
        --records \
'"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"' \
'"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"' \
'"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"'

应该可以正常工作，最多 500 条记录或命令行缓冲区的限制。

结果将类似于：

{
  "FailedPutCount": 0, 
  "RequestResponses": [
    {
      "RecordId": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQgZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29tbW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxhIHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lkZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlkIGVzdCBsYWJvcnVtLg=="
    }, 
    {
      "RecordId": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQgZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29tbW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxhIHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lkZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlkIGVzdCBsYWJvcnVtLg=="
    }, 
    {
      "RecordId": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQgZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29tbW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxhIHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lkZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlkIGVzdCBsYWJvcnVtLg=="
    }
  ]
}

Answer 2

我可以通过以下方式解决这个问题。

aws firehose put-record-batch --delivery-stream-name S3TestingStream --records '"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"' \
'"{\"productid\":2,\"productname\":\"bus\",\"productprice\":\"5.40\"}"' \
'"{\"productid\":3,\"productname\":\"car\",\"productprice\":\"3.20\"}"'

这给了我成功 putrecord 后的记录 ID。

   "FailedPutCount": 0, 
"RequestResponses": [
    {
        "RecordId": "N3w6uvXwuxbd9sdXrNI3Kcptcwu5j/oco1wuIjq6vrbz+HAkDEw0N1Rah58pwx6h5dZozBScJ4EFFTQf0SxhH5A0Ie6LGdfJ0OuStB3zb512a2O/bmQretZWGopN3bvzMQO3EIZ3ksNaM63rCIbIT5fI6ZkYzX+rLu8QQtaMCS/881z6qwMGG7n3eJCtWK36uTs4M0X5hEpR1+j8B4hyjdXexu212f4j"
    }, 
    {
        "RecordId": "eWW4ZX9bwyIjVdJ/TQbb+Yh+vJFo/O/CdeT4MXby4T4DNydjoGgoPluQFJbwzFMV2Q6OPWXJblNgjRG3nGVGn73WPQC3AhqWQ3Bqc1ioa2E5yyYGEK1bbk57I4lFN9cdhkkC01POKwz9RszZLkJB38iZsIK4Gh2lp2p65LTPmBWdP0c8Hgd6WOJfrX8IHRhLhc00tuBBi6IM5jCdWRAyLhf45fGhPTee"
    }, 
    {
        "RecordId": "vf8NVYzfGQiCPX0ocRhdhqeTlobYv9FIOacHanqCrkbspHM71Zg36rS5YLu5clIvCSIs6cX6EnaStn5L6s8LgpGQNBBfgr5zk9ydRWNHHF3kkEvqZnPcHU64R4KpkvVhqDTVELqMsFxPfPFEP9Gikzc3d3IqN2EMc+T8fAj7g66nNZgQ3TF3y0SMeZpGyrezDYfLUzGuP/ZzbhCvMr2ypRW51SszGUBN"
    }
]

}