无法获取行中数组 <struct<string, string>> 的 avro 数据
unable to get avro data for array<struct<string, string>> in rows
您好,我有一个带有结构数组的 avro 模式,我可以将数据保存为 avro。但在从
检索数据时
array<struct<string, string>>
我无法排队。我在单行中获取的所有数据。
这里是 table 定义
CREATE EXTERNAL TABLE meterevents ROW FORMAT SERDE org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED as INPUTFORMAT org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION '/......' TBLPROPERTIES ('avro.schema.url'='/..../schema.avsc');
配置单元table结构
nametype struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>> from deserializer
names struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>> from deserializer
enddeviceeventdetails struct<enddeviceeventdetailsname:string,enddeviceeventdetailsvalue:string> from deserializer
enddeviceevent struct<mrid:string,createddatetime:string,issuerid:string,issuertrackingid:string,reason:string,severity:string,userid:string,asset:struct<assetmrid:string,assetnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>,enddeviceeventdetails:array<struct<enddeviceeventdetailsname:string,enddeviceeventdetailsvalue:string>>,enddeviceeventtype:string,enddeviceeventnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>,status:struct<statusdatetime:string,statusreason:string,statusremark:string,statusvalue:string>,usagepoint:struct<usagepointmrid:string,usagepointnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>> from deserializer
enddeviceeventtype struct<enddeviceeventtypemrid:string,enddeviceeventtypedomain:string,enddeviceeventtypeeventoraction:string,enddeviceeventtypesubdomain:string,type:string,enddeviceeventtypenames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>> from deserializer
header struct<noun:string,context:string,verb:string,value:string,source:string,timestamp:string,correlationid:string,name:string,messageid:string,property:struct<propertyname:array<string>,propertyvalue:array<string>>> from deserializer
payload struct<enddeviceevents:array<struct<mrid:string,createddatetime:string,issuerid:string,issuertrackingid:string,reason:string,severity:string,userid:string,asset:struct<assetmrid:string,assetnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>,enddeviceeventdetails:array<struct<enddeviceeventdetailsname:string,enddeviceeventdetailsvalue:string>>,enddeviceeventtype:string,enddeviceeventnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>,status:struct<statusdatetime:string,statusreason:string,statusremark:string,statusvalue:string>,usagepoint:struct<usagepointmrid:string,usagepointnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>>>,enddeviceeventtype:array<struct<enddeviceeventtypemrid:string,enddeviceeventtypedomain:string,enddeviceeventtypeeventoraction:string,enddeviceeventtypesubdomain:string,type:string,enddeviceeventtypenames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>>>
我在查询中使用 "LATERAL VIEW explode" 选项
select eddetails.enddeviceeventdetailsname, eddetails.enddeviceeventdetailsvalue
FROM meterevents_tmp
LATERAL VIEW explode(payload.enddeviceevents.enddeviceeventdetails) ed AS eddetails
limit 1;
但我仍然在单行中获取数据。
enddeviceeventdetailsname enddeviceeventdetailsvalue
["EventSequenceNumber","EventSequenceNumber","EventSequenceNumber","EventSequenceNumber"] ["683","684","685","686"
我想将此数据设为
enddeviceeventdetailsname enddeviceeventdetailsvalue
EventSequenceNumber 683
EventSequenceNumber 684
EventSequenceNumber 685
EventSequenceNumber 686
我已经阅读了 Whosebug 上的另一个问题:Exploding Array of Struct using HiveQL
但是无法获得预期的输出。因为在那 post 它的蜂巢外部 table 而不是我无法指定 "MAP KEYS TERMINATED BY" 和 "COLLECTION ITEMS TERMINATED BY"
的 serde
非常感谢任何帮助。
谢谢
我能够解决这个问题 ---
我无法按行获取输出,因为
array<struct<string,string>>
是父数组的一部分
array<struct<array<struct<string, string>>>
我更新了查询并使用了嵌套展开
select eddetails.enddeviceeventdetailsname, eddetails.enddeviceeventdetailsvalue from (select ede.enddeviceeventdetails FROM meterevents_tmp LATERAL VIEW explode(payload.enddeviceevents) e AS ede) t LATERAL VIEW explode(t.enddeviceeventdetails) ed AS eddetails limit 10;
我得到了想要的输出 -
enddeviceeventdetailsname enddeviceeventdetailsvalue
EventSequenceNumber 683
EventSequenceNumber 684
EventSequenceNumber 685
EventSequenceNumber 686
您好,我有一个带有结构数组的 avro 模式,我可以将数据保存为 avro。但在从
检索数据时array<struct<string, string>>
我无法排队。我在单行中获取的所有数据。
这里是 table 定义
CREATE EXTERNAL TABLE meterevents ROW FORMAT SERDE org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED as INPUTFORMAT org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION '/......' TBLPROPERTIES ('avro.schema.url'='/..../schema.avsc');
配置单元table结构
nametype struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>> from deserializer
names struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>> from deserializer
enddeviceeventdetails struct<enddeviceeventdetailsname:string,enddeviceeventdetailsvalue:string> from deserializer
enddeviceevent struct<mrid:string,createddatetime:string,issuerid:string,issuertrackingid:string,reason:string,severity:string,userid:string,asset:struct<assetmrid:string,assetnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>,enddeviceeventdetails:array<struct<enddeviceeventdetailsname:string,enddeviceeventdetailsvalue:string>>,enddeviceeventtype:string,enddeviceeventnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>,status:struct<statusdatetime:string,statusreason:string,statusremark:string,statusvalue:string>,usagepoint:struct<usagepointmrid:string,usagepointnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>> from deserializer
enddeviceeventtype struct<enddeviceeventtypemrid:string,enddeviceeventtypedomain:string,enddeviceeventtypeeventoraction:string,enddeviceeventtypesubdomain:string,type:string,enddeviceeventtypenames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>> from deserializer
header struct<noun:string,context:string,verb:string,value:string,source:string,timestamp:string,correlationid:string,name:string,messageid:string,property:struct<propertyname:array<string>,propertyvalue:array<string>>> from deserializer
payload struct<enddeviceevents:array<struct<mrid:string,createddatetime:string,issuerid:string,issuertrackingid:string,reason:string,severity:string,userid:string,asset:struct<assetmrid:string,assetnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>,enddeviceeventdetails:array<struct<enddeviceeventdetailsname:string,enddeviceeventdetailsvalue:string>>,enddeviceeventtype:string,enddeviceeventnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>,status:struct<statusdatetime:string,statusreason:string,statusremark:string,statusvalue:string>,usagepoint:struct<usagepointmrid:string,usagepointnames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>>>,enddeviceeventtype:array<struct<enddeviceeventtypemrid:string,enddeviceeventtypedomain:string,enddeviceeventtypeeventoraction:string,enddeviceeventtypesubdomain:string,type:string,enddeviceeventtypenames:array<struct<name:string,nametype:struct<nametypedescription:string,nametypename:string,nametypeauthority:struct<nametypeauthorityname:string,nametypeauthoritydescription:string>>>>>>>
我在查询中使用 "LATERAL VIEW explode" 选项
select eddetails.enddeviceeventdetailsname, eddetails.enddeviceeventdetailsvalue
FROM meterevents_tmp
LATERAL VIEW explode(payload.enddeviceevents.enddeviceeventdetails) ed AS eddetails
limit 1;
但我仍然在单行中获取数据。
enddeviceeventdetailsname enddeviceeventdetailsvalue
["EventSequenceNumber","EventSequenceNumber","EventSequenceNumber","EventSequenceNumber"] ["683","684","685","686"
我想将此数据设为
enddeviceeventdetailsname enddeviceeventdetailsvalue
EventSequenceNumber 683
EventSequenceNumber 684
EventSequenceNumber 685
EventSequenceNumber 686
我已经阅读了 Whosebug 上的另一个问题:Exploding Array of Struct using HiveQL
但是无法获得预期的输出。因为在那 post 它的蜂巢外部 table 而不是我无法指定 "MAP KEYS TERMINATED BY" 和 "COLLECTION ITEMS TERMINATED BY"
的 serde非常感谢任何帮助。
谢谢
我能够解决这个问题 ---
我无法按行获取输出,因为
array<struct<string,string>>
是父数组的一部分
array<struct<array<struct<string, string>>>
我更新了查询并使用了嵌套展开
select eddetails.enddeviceeventdetailsname, eddetails.enddeviceeventdetailsvalue from (select ede.enddeviceeventdetails FROM meterevents_tmp LATERAL VIEW explode(payload.enddeviceevents) e AS ede) t LATERAL VIEW explode(t.enddeviceeventdetails) ed AS eddetails limit 10;
我得到了想要的输出 -
enddeviceeventdetailsname enddeviceeventdetailsvalue
EventSequenceNumber 683
EventSequenceNumber 684
EventSequenceNumber 685
EventSequenceNumber 686