AWS Athena 将结构数组导出到 JSON

Question

我有一个 Athena table，其中一些字段具有相当复杂的嵌套格式。 S3 中的支持记录是 JSON。按照这些思路（但我们还有更多层级的嵌套）：

CREATE EXTERNAL TABLE IF NOT EXISTS test (
  timestamp double,
  stats array<struct<time:double, mean:double, var:double>>,
  dets array<struct<coords: array<double>, header:struct<frame:int, 
    seq:int, name:string>>>,
  pos struct<x:double, y:double, theta:double>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ('ignore.malformed.json'='true')
LOCATION 's3://test-bucket/test-folder/'

现在我们需要能够查询数据并将结果导入Python进行分析。由于安全限制，我无法直接连接到 Athena；我需要能够向某人提供查询，然后他们会给我 CSV 结果。

如果我们直接执行 select *，我们会以不完全 JSON 的格式返回 struct/array 列。这是一个示例输入文件条目：

{"timestamp":1520640777.666096,"stats":[{"time":15,"mean":45.23,"var":0.31},{"time":19,"mean":17.315,"var":2.612}],"dets":[{"coords":[2.4,1.7,0.3], "header":{"frame":1,"seq":1,"name":"hello"}}],"pos": {"x":5,"y":1.4,"theta":0.04}}

示例输出：

select * from test

"timestamp","stats","dets","pos"
"1.520640777666096E9","[{time=15.0, mean=45.23, var=0.31}, {time=19.0, mean=17.315, var=2.612}]","[{coords=[2.4, 1.7, 0.3], header={frame=1, seq=1, name=hello}}]","{x=5.0, y=1.4, theta=0.04}"

我希望以更方便的格式导出这些嵌套字段 - 以 JSON 格式导出它们会很棒。

不幸的是，转换为 JSON 似乎仅适用于映射，不适用于结构，因为它只是将所有内容展平为数组：

SELECT timestamp, cast(stats as JSON) as stats, cast(dets as JSON) as dets, cast(pos as JSON) as pos FROM "sampledb"."test"

"timestamp","stats","dets","pos"
"1.520640777666096E9","[[15.0,45.23,0.31],[19.0,17.315,2.612]]","[[[2.4,1.7,0.3],[1,1,""hello""]]]","[5.0,1.4,0.04]"

是否有转换为 JSON（或其他易于导入的格式）的好方法，还是我应该继续执行自定义解析功能？

Answer 1

我浏览了所有文档，不幸的是，目前似乎没有办法做到这一点。唯一可能的解决方法是

converting a struct to a json when querying athena

SELECT
  my_field,
  my_field.a,
  my_field.b,
  my_field.c.d,
  my_field.c.e
FROM 
  my_table

或者我会使用 post 处理将数据转换为 json。下面的脚本显示了如何

#!/usr/bin/env python
import io
import re

pattern1 = re.compile(r'(?<={)([a-z]+)=', re.I)
pattern2 = re.compile(r':([a-z][^,{}. [\]]+)', re.I)
pattern3 = re.compile(r'\"', re.I)

with io.open("test.csv") as f:
    headers = list(map(lambda f: f.strip(), f.readline().split(",")))
    for line in f.readlines():
        orig_line = line
        data = []
        for i, l in enumerate(line.split('","')):
            data.append(headers[i] + ":" + re.sub('^"|"$', "", l))

        line = "{" + ','.join(data) + "}"
        line = pattern1.sub(r'"":', line)
        line = pattern2.sub(r':""', line)
        print(line)

输入数据的输出是

{"timestamp":1.520640777666096E9,"stats":[{"time":15.0, "mean":45.23, "var":0.31}, {"time":19.0, "mean":17.315, "var":2.612}],"dets":[{"coords":[2.4, 1.7, 0.3], "header":{"frame":1, "seq":1, "name":"hello"}}],"pos":{"x":5.0, "y":1.4, "theta":0.04}
}

哪个是有效的JSON

Answer 2

@tarun 的 python 代码几乎让我到达了那里，但由于我的数据，我不得不以多种方式修改它。特别是，我有：

json 在 Athena 中保存为字符串的结构
包含多个单词的字符串，因此需要用双引号括起来。其中一些包含“[]”和“{}”符号。

这是对我有用的代码，希望对其他人有用：

#!/usr/bin/env python
import io
import re, sys

pattern1 = re.compile(r'(?<={)([a-z]+)=', re.I)
pattern2 = re.compile(r':([a-z][^,{}. [\]]+)', re.I)
pattern3 = re.compile(r'\"', re.I)

with io.open(sys.argv[1]) as f:
    headers = list(map(lambda f: f.strip(), f.readline().split(",")))
    print(headers)
    for line in f.readlines():

        orig_line = line
        #save the double quote cases, which mean there is a string with quotes inside
        line = re.sub('""', "#", orig_line)
        data = []
        for i, l in enumerate(line.split('","')):
            item = re.sub('^"|"$', "", l.rstrip())
            if (item[0] == "{" and item[-1] == "}") or (item[0] == "[" and item[-1] == "]"):
                data.append(headers[i] + ":" + item)
            else: #we have a string
                data.append(headers[i] + ": \"" + item + "\"")

        line = "{" + ','.join(data) + "}"
        line = pattern1.sub(r'"":', line)
        line = pattern2.sub(r':""', line)

        #restate the double quotes to single ones, once inside the json
        line = re.sub("#", '"', line)

        print(line)

Answer 3

此方法不是通过修改Query。

它由 Post 处理对于 Javascript/Nodejs 我们可以使用 npm 包 athena-struct-parser.

带示例的详细答案

参考 - https://www.npmjs.com/package/athena-struct-parser

Answer 4

我使用了一种简单的方法来绕过结构 -> json Athena 限制。我创建了第二个 table，其中 json 列被保存为原始字符串。使用 presto json 和数组函数，我能够查询数据和 return 有效的 json 字符串到我的程序：

--Array transform functions too
select 
  json_extract_scalar(dd, '$.timestamp') as timestamp,
  transform(cast(json_extract(json_parse(dd), '$.stats') as ARRAY<JSON>), x -> json_extract_scalar(x, '$.time')) as arr_stats_time,
  transform(cast(json_extract(json_parse(dd), '$.stats') as ARRAY<JSON>), x -> json_extract_scalar(x, '$.mean')) as arr_stats_mean,
  transform(cast(json_extract(json_parse(dd), '$.stats') as ARRAY<JSON>), x -> json_extract_scalar(x, '$.var')) as arr_stats_var
from 
(select '{"timestamp":1520640777.666096,"stats":[{"time":15,"mean":45.23,"var":0.31},{"time":19,"mean":17.315,"var":2.612}],"dets":[{"coords":[2.4,1.7,0.3], "header":{"frame":1,"seq":1,"name":"hello"}}],"pos": {"x":5,"y":1.4,"theta":0.04}}' as dd);

我知道执行查询需要更长的时间，但有一些方法可以优化。

Answer 5

我通过使用相同的 S3 位置创建第二个 table 来解决这个问题，但将字段的数据类型更改为字符串。然后生成的 CSV 包含 Athena 从 JSON 文件中的对象中提取的字符串，我能够解析结果。

AWS Athena 将结构数组导出到 JSON

AWS Athena export array of structs to JSON

json

presto

amazon-athena