如何将字符串解析为 HIVE 中的映射数组

Question

我有一个从系统日志中提取的配置单元 table。数据以一种奇怪的格式（映射数组）编码，其中数组的每个元素都包含 field_name 和 value。列类型为 STRING。就像下面的例子：

select 1 as user_id, '[{"field":"name", "value":"Bob"}, {"field":"gender", "value":"M"}]' as user_info
union all
select 2 as user_id, '[{"field":"gender", "value":"F"}, {"field":"age", "value":22}, {"field":"name", "value":"Ana"}]' as user_info;

它创建了这样的东西：

user_id	user_info
1	[{"field":"name", "value":"Bob"}, {"field":"gender", "value":"M"}]
2	[{"field":"gender", "value":"F"}, {"field":"age", "value":22}, {"field":"name", "value":"Ana"}]

请注意，数组大小并不总是相同。我正在尝试将地图数组转换为简单地图。然后，这就是我期望的结果：

user_id	user_info
1	{"name":"Bob", "gender":"M"}
2	{"name":"Ana", "gender":"F", "age":22}

我计划通过 3 个步骤实现：(1) 解析字符串列以创建映射数组，(2) 分解数组（使用横向视图），(3) 收集字段列表，以及按 user_id

分组

我正在努力完成第一步：解析字符串列以创建映射数组。任何帮助将不胜感激：D

Answer 1

查看代码中的注释。要转换为映射的字符串数组由此 split(user_info, '(?<=\}) *, *(?=\{)') 生成。然后分解，每个元素转化为map。

with mydata as
(select 1 as user_id, '[{"field":"name", "value":"Bob"}, {"field":"gender", "value":"M"}]' as user_info
union all
select 2 as user_id, '[{"field":"gender", "value":"F"}, {"field":"age", "value":22}, {"field":"name", "value":"Ana"}]' as user_info
)

select user_id,
       --build new map
       str_to_map(concat('name:', name, nvl(concat(',','gender:', gender),''),  nvl(concat(',','age:', age),'') )) as user_info
from 
(
select user_id, 
      --get name, gender, age, aggregate by user_id
      max(case when user_info['field'] = 'name' then user_info['value'] end) name,
      max(case when user_info['field'] = 'gender' then user_info['value'] end) gender,
      max(case when user_info['field'] = 'age' then user_info['value'] end) age
      
from      
(
select s.user_id, 
       --remove {} and ", convert to map
       str_to_map(regexp_replace(e.element,'^\{| *"|\}$','')) as user_info 
from
(
select user_id, regexp_replace(user_info, '^\[|\]$','') as user_info -- remove []
 from mydata
)s lateral view outer explode(split(user_info, '(?<=\}) *, *(?=\{)'))e as element --split by comma between }{ with optional spaces in between
) s
group by user_id
)s

结果：

user_id   user_info 
1        {"name":"Bob","gender":"M"}
2        {"name":"Ana","gender":"F","age":"22"}

如何将字符串解析为 HIVE 中的映射数组

How to parse an string to an array of maps in HIVE

sql

json

dictionary

hive

hiveql