Hive 上的 CROSS APPLY SQL 服务器查询

Question

HDP-2.5.0.0 使用 Ambari 2.4.0.1

HivetableReportSetting如下：

id int

serializedreportsetting String

列 'serializedreportsetting' 是源 SQL 服务器数据库中的 XML 数据类型，但在 Sqoop 导入期间被转换为字符串，这是它在 [=55= 中的样子] 服务器：

<ReportSettings4 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Trigger>
  <Manual>true</Manual>
  </Trigger>
<StartTime>
    <Year>8</Year>
    <Month>1</Month>
    <Day>1</Day>
    <Hour>0</Hour>
    <Minute>0</Minute>
  </StartTime>
  <ReportPeriod>
    <Month>0</Month>
    <Day>0</Day>
    <Hour>0</Hour>
    <Minute>5</Minute>
  </ReportPeriod>
  <Theft>
    <DigitalInput>true</DigitalInput>
    <Can>false</Can>
  </Theft>
  <SequenceNo>0</SequenceNo>
</ReportSettings4>

在蜂巢中 table :

<ReportSettings4 xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><Trigger><Manual>true</Manual></Trigger><StartTime><Year>8</Year><Month>12</Month><Day>31</Day><Hour>23</Hour><Minute>34</Minute></StartTime><ReportPeriod><Month>0</Month><Day>0</Day><Hour>4</Hour><Minute>0</Minute></ReportPeriod><Theft><DigitalInput>false</DigitalInput><Can>false</Can></Theft><SequenceNo>3</SequenceNo></ReportSettings4>

在 SQL 服务器上运行良好的查询：

SELECT
r.VehicleId
,rs.value('(Trigger/Manual)[1]', 'bit') AS RS_Trigger_Manual, ,CAST(CONCAT(CASE WHEN rs.value('(StartTime/Year)[1]', 'int') < 10 THEN CONCAT('200',rs.value('(StartTime/Year)[1]', 'int')) ELSE CONCAT('20',rs.value('(StartTime/Year)[1]', 'int')) END,'-',rs.value('(StartTime/Month)[1]', 'int'),'-',rs.value('(StartTime/Day)[1]', 'int'),' ',rs.value('(StartTime/Hour)[1]', 'int'),':',rs.value('(StartTime/Minute)[1]', 'int'),':','00.000') AS datetime) AS RS_StartTime
,rs.value('(ReportPeriod/Month)[1]', 'int') AS RS_ReportPeriod_Month
,rs.value('(ReportPeriod/Day)[1]', 'int') AS RS_ReportPeriod_Day
,rs.value('(ReportPeriod/Hour)[1]', 'int') AS RS_ReportPeriod_Hour
,rs.value('(ReportPeriod/Minute)[1]', 'int') AS RS_ReportPeriod_Minute
,rs.value('(Theft/DigitalInput)[1]', 'bit') AS RS_Theft_DigitalInput
,rs.value('(Theft/Can)[1]', 'bit') AS RS_Theft_Can,rs.value('(SequenceNo)[1]', 'int') 

AS RS_SequenceNo FROM ReportSetting r
  CROSS APPLY SerializedReportSetting.nodes('/*') AS ReportSettings(rs)

我可以想到 of/do 以下 :

要使用CROSS APPLY，我想需要使用lateral view，这里我没有将serializedreportsetting 作为数组，所以explode() 将不起作用。有人可以验证我的思考方向是否正确吗
我只是尝试使用built-in xpath udf将serializedreportsetting中的数据作为列获取，但是，我没有得到任何记录，少数试验如下：

select xpath(SerializedReportSetting,'/*') 来自 ReportSetting 限制 1;

select xpath(SerializedReportSetting,'/ReportSettings4') 来自 ReportSetting 限制 1;

select xpath(SerializedReportSetting,'/Trigger/Manual') 来自 ReportSetting 限制 1;

**********UPDATE-1************

我使用 regexp_replace 来应对上述挑战：

SELECT id,
  xpath_string(SerializedReportSetting,'/ReportSettings/Trigger/Manual')        AS RS_Trigger_Manual,
  xpath_string(SerializedReportSetting,'/ReportSettings/Trigger/DriveChange')   AS RS_Trigger_DriveChange
FROM
  (SELECT id,
    regexp_replace(SerializedReportSetting, 'ReportSettings+\d','ReportSettings') AS SerializedReportSetting
  FROM reportsetting
  WHERE id IN (1701548,3185,1700231,1700232)
  ) reportsetting_regex;

Answer 1

在 xpath 中，他们明确表示：

The xpath() function always returns a hive array of strings. If the expression results in a non-text value (e.g., another xml node) the function will return an empty array

因此您可以使用：xpath(SerializedReportSetting,'/ReportSettings4/Trigg‌er/Manual/text()') from ReportSetting limit 1;

或者更好的选择是使用 xpath_boolean/xpath_int:

xpath_boolean - Returns true if the XPath expression evaluates to true, or if a matching node is found.

xpath_boolean(SerializedReportSetting,'/ReportSettings4/Trigg‌er/Manual') from ReportSetting limit 1;

xpath_short, xpath_int, xpath_long These functions return an integer numeric value, or the value zero if no match is found, or a match is found but the value is non-numeric. Mathematical operations are supported. In cases where the value overflows the return type, then the maximum value for the type is returned.

xpath_int(SerializedReportSetting,'/ReportSettings4/ReportPeriod/Month') from ReportSetting limit 1;

Hive 上的 CROSS APPLY SQL 服务器查询

CROSS APPLY SQL Server query on Hive

hadoop

hive

hortonworks-data-platform

hive-udf