在 Hive 中转置数据

Question

我有一个 Hive table，数据格式如下

day       class  start_time  count  kpi1  kpi2  kpi3  kpi4  ...  kpi160
-----------------------------------------------------------------------
20161010  abc    00          12     1     0     null  0     ...

我想编写一个配置单元查询来获取以下格式的数据使用一些计算，如 max、min 和 avg.

day       class  start_time  count  kpi_name  kpi_max   kpi_min   kpi_avg
-----------------------------------------------------------------------
20161010  abc    00          12     kpi1      max(kpi1) min(kpi1) avg(kpi1)
20161010  abc    00          12     kpi2      max(kpi2) min(kpi2) avg(kpi2)

请提出一个解决方案来获取这种格式的数据。

谢谢。

Answer 1

如果你想得到 min,max,avg ，你必须指定按列分组，假设你想按天分组。

SELECT day,
       class,
       start_time,
       count,
       kpi1,
       MAX(kpi1) as max_kpi1,
       MIN(kpi1) as min_kpi1,
       AVG(kpi1) as avg_kpi1

FROM table

GROUP BY day

Answer 2

你需要把所有的kpi放在一张地图上，分解地图创建一列，然后聚合。

示例：

数据:

+---------+------+-----------+-------+-----+-----+------+------+------+------+
|day_     |class |start_time |count_ |kpi0 |kpi1 | kpi2 | kpi3 | kpi4 | kpi5 |
+---------+------+-----------+-------+-----+-----+------+------+------+------+
|20161010 |abc   |00         |12     |1    |2    |3     |8     |9     |6     |
+---------+------+-----------+-------+-----+-----+------+------+------+------+
|20161010 |abc   |00         |12     |4    |5    |null  |6     |10    |null  |
+---------+------+-----------+-------+-----+-----+------+------+------+------+

查询:

SELECT day_
  , class
  , start_time
  , count_
  , kpi_type
  , MAX(vals) AS max_vals
  , MIN(vals) AS min_vals
  , AVG(vals) AS avg_vals
FROM (
  SELECT day_, class, start_time, count_, kpi_type, vals
  FROM database.table
  LATERAL VIEW EXPLODE(MAP('kpi0', kpi0
                         , 'kpi1', kpi1
                         , 'kpi2', kpi2
                         , 'kpi3', kpi3
                         , 'kpi4', kpi4
                         , 'kpi5', kpi5)) et AS kpi_type, vals ) x
GROUP BY day_, class_, start_time, count_, kpi_type

输出:

+---------+------+-----------+-------+---------+---------+---------+---------+
|day_     |class |start_time |count_ |kpi_type |max_vals |min_vals |avg_vals |
+---------+------+-----------+-------+---------+---------+---------+---------+
|20161010 |abc   |00         |12     |kpi0     |4        |1        |2.5      | 
+---------+------+-----------+-------+---------+---------+---------+---------+
|20161010 |abc   |00         |12     |kpi1     |5        |2        |3.5      |
+---------+------+-----------+-------+---------+---------+---------+---------+
|20161010 |abc   |00         |12     |kpi2     |3        |3        |3.0      |
+---------+------+-----------+-------+---------+---------+---------+---------+
|20161010 |abc   |00         |12     |kpi3     |8        |6        |7.0      |
+---------+------+-----------+-------+---------+---------+---------+---------+
|20161010 |abc   |00         |12     |kpi4     |10       |9        |9.5      |
+---------+------+-----------+-------+---------+---------+---------+---------+
|20161010 |abc   |00         |12     |kpi5     |6        |6        |6.0      |
+---------+------+-----------+-------+---------+---------+---------+---------+

在 Hive 中转置数据

Transposing data in Hive

hive

hiveql