如何解码来自 kinesis firehose 的流数据？

Question

我已经设置了一个 Kinesis firebase 以将 cloudwatch 日志流式传输到 http 端点。 http 端点是一个 nodejs 应用程序。我禁用了 zip，在 kinesis firehose 中编码。

我可以在节点应用程序中接收它，但我不知道如何解码记录。

我收到的其中一条记录格式如下。它不是 base64 字符串。我不知道它是什么编码格式。我该如何解码？

 [
    {
      data: 'H4sIAAAAAAAAAK1QXWvbMBT9K8bsMZl19XEl5c0lbiks24i9vTRhyJZSDHaS2cq6rvS/9zrroBT6UBgCCc7RvefjIe3DOLrbUN0fQ7pIl3mV/1gVZZlfFeksPdztw0AwRwmMMZCSKYK7w+3VcDgdicnc3Zh1rq+9y2IY41+2jENw/TTIOGQMM26zmw+f8qooq+1OCe9c0+DOg7S2MXZnla1rL2rJnQFaMZ7qsRnaY2wP+8u2i2EY08XNtHlMt2eF4lfYxwl8SFtPQgK50EwaQGsks6AZt6C4klqD5XRLxVAywVCB1VygYBRKcBKLLXUQXU9xALm0NCSMRJz964bWl1W+rpJ1+Hmir9d+kYRGAnd8N0evm7n0wsyd0zj3GiCgVN44mXwn35RgkTxH3+zTx9lrw5orIywzaIzSYEAKVHSQPADQKxmTQitrEDQyeMuwMval4eLz8r12/4O7N+t85W5dfP3y/j43cXkaXDw3Cpx95Crpx028aLsu+OQlB2diFfrDcJ+U7Z8wgSZZXRDofifPxLcxkDLiGZ/Sbx+fALQcXhYQAwAA'
    }
...

Answer 1

它是 gzip base64:

data="H4sIAAAAAAAAAK1QXWvbMBT9K8bsMZl19XEl5c0lbiks24i9vTRhyJZSDHaS2cq6rvS/9zrroBT6UBgCCc7RvefjIe3DOLrbUN0fQ7pIl3mV/1gVZZlfFeksPdztw0AwRwmMMZCSKYK7w+3VcDgdicnc3Zh1rq+9y2IY41+2jENw/TTIOGQMM26zmw+f8qooq+1OCe9c0+DOg7S2MXZnla1rL2rJnQFaMZ7qsRnaY2wP+8u2i2EY08XNtHlMt2eF4lfYxwl8SFtPQgK50EwaQGsks6AZt6C4klqD5XRLxVAywVCB1VygYBRKcBKLLXUQXU9xALm0NCSMRJz964bWl1W+rpJ1+Hmir9d+kYRGAnd8N0evm7n0wsyd0zj3GiCgVN44mXwn35RgkTxH3+zTx9lrw5orIywzaIzSYEAKVHSQPADQKxmTQitrEDQyeMuwMval4eLz8r12/4O7N+t85W5dfP3y/j43cXkaXDw3Cpx95Crpx028aLsu+OQlB2diFfrDcJ+U7Z8wgSZZXRDofifPxLcxkDLiGZ/Sbx+fALQcXhYQAwAA"

import base64
import gzip

decodedBytes = base64.b64decode(data)
print(gzip.decompress(decodedBytes))

给出：

b'{"messageType":"DATA_MESSAGE","owner":"264100014405","logGroup":"/aws/lambda/test","logStream":"2021/06/29/[$LATEST]f53daacc6fd1499c89f959bbd3b42a81","subscriptionFilters":["logs"],"logEvents":[{"id":"36237048169840917029152547719247745064030651972363026432","timestamp":1624925438466,"message":"START RequestId: ec412a2f-6d7c-4d38-aa76-d711e645d8a4 Version: $LATEST\n"},{"id":"36237048172583908688571814365656638411566400437598617601","timestamp":1624925438589,"message":"END RequestId: ec412a2f-6d7c-4d38-aa76-d711e645d8a4\n"},{"id":"36237048172583908688571814365656638411566400437598617602","timestamp":1624925438589,"message":"REPORT RequestId: ec412a2f-6d7c-4d38-aa76-d711e645d8a4\tDuration: 120.25 ms\tBilled Duration: 121 ms\tMemory Size: 128 MB\tMax Memory Used: 66 MB\t\n"}]}'

如何解码来自 kinesis firehose 的流数据？

How can I decode stream data from kinesis firehose?

amazon-web-services

amazon-kinesis

amazon-kinesis-firehose