多重全外连接（来自同一 table 中的数据）

Question

我有一些日志数据如下所示

日志

+---------------------+---------+---------+------+
|        date         | api_key | version | data |
+---------------------+---------+---------+------+
| 2018-05-08 01:00:00 | AAA     | v1      | data |
| 2018-05-08 02:00:00 | AAA     | v2      | data |
| 2018-05-06 03:00:00 | AAA     | v2      | data |
| 2018-05-06 04:00:00 | BBB     | v1      | data |
+---------------------+---------+---------+------+

date 是 API 调用的日期
api_key 是 API 使用的键
version是使用的API的版本（我们有两个）

此数据目前位于 RDBMS 中，我需要将数据移动到 Athena。

聚合

有一些遗留代码会定期运行以聚合日志。然后，此聚合数据显示在仪表板上。本质上，它将平面日志数据（上图）转换为由 API 键聚合的数据，给出了基于版本的调用计数，以及在不同时间 windows（所有时间，1 天，7 天，等等）

例如，以下聚合 table 显示聚合的原始数据。

+---------+------+---------+---------+--------+-----------+-----------+--------+-----------+-----------+
| api_key | hits | hits_v1 | hits_v2 | hits_1 | hits_1_v1 | hits_1_v2 | hits_7 | hits_7_v1 | hits_7_v2 |
+---------+------+---------+---------+--------+-----------+-----------+--------+-----------+-----------+
| AAA     |    3 |       1 |       2 |      2 |         1 |         1 |      3 |         1 |         2 |
| BBB     |    1 |       1 |       0 |      0 |         0 |         0 |      1 |         1 |         0 |
+---------+------+---------+---------+--------+-----------+-----------+--------+-----------+-----------+

这些是列的含义：

hits给定api-key
hits_v1 v1

api-key

hits_v2 v2

api-key

hits_1 给定 api-key
hits_1_v1 给定 api-key

v1

hits_1_v2 给定 api-key

v2

hits_7 给定 api-key
hits_7_v1 给定 api-key

v1

hits_7_v2 给定 api-key

v2

SQL

下面是我用来创建这个聚合的 SQL。table。

SELECT coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key, hits_1_v2.api_key) api_key,
    coalesce(hits_v1.hits_v1,0) + coalesce(hits_v2.hits_v2,0) hits, 
    coalesce(hits_v1.hits_v1,0) hits_v1, 
    coalesce(hits_v2.hits_v2,0) hits_v2,

    coalesce(hits_1_v1.hits_1_v1,0) + coalesce(hits_1_v2.hits_1_v2,0) hits_1, 
    coalesce(hits_1_v1.hits_1_v1,0) hits_1_v1, 
    coalesce(hits_1_v2.hits_1_v2,0) hits_1_v2,

    coalesce(hits_7_v1.hits_7_v1,0) + coalesce(hits_7_v2.hits_7_v2,0) hits_7, 
    coalesce(hits_7_v1.hits_7_v1,0) hits_7_v1, 
    coalesce(hits_7_v2.hits_7_v2,0) hits_7_v2
FROM 
(
    (select api_key,count(*) as hits_v1 from logs where (version='v1' or version='') group by api_key) hits_v1
    FULL OUTER JOIN
    (select api_key,count(*) as hits_v2 from logs where version='v2' group by api_key) hits_v2                                                                       on hits_v2.api_key   = hits_v1.api_key

    FULL OUTER JOIN 
    (select api_key,count(*) as hits_1_v1 from logs where (version='v1' or version='') and (date > localtimestamp - interval '1' day) group by api_key) hits_1_v1    on hits_1_v1.api_key = coalesce(hits_v1.api_key, hits_v2.api_key)
    FULL OUTER JOIN 
    (select api_key,count(*) as hits_1_v2 from logs where version='v2' and (date > localtimestamp - interval '1' day) group by api_key) hits_1_v2                    on hits_1_v2.api_key = coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key)

    FULL OUTER JOIN 
    (select api_key,count(*) as hits_7_v1 from logs where (version='v1' or version='') and (date > localtimestamp - interval '7' day) group by api_key) hits_7_v1    on hits_7_v1.api_key = coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key, hits_1_v2.api_key)
    FULL OUTER JOIN 
    (select api_key,count(*) as hits_7_v2 from logs where version='v2' and (date > localtimestamp - interval '7' day) group by api_key) hits_7_v2                    on hits_7_v2.api_key = coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key, hits_1_v2.api_key, hits_7_v1.api_key)
)
order by api_key asc

如您所见，它非常重复且冗长。更糟糕的是，我需要添加的列越多（14 天、30 天、60 天等），我每次都需要添加到 on 子句以包括所有以前的连接。

这行得通，但我相信一定有更简洁的方法来做到这一点。有人可以帮忙吗？

PS。是的，我确实需要保留这个聚合 table - 它可能不太好，但是一大堆其他代码都依赖于它，所以它不能改变。

Answer 1

使用条件聚合：

select api_key,
       sum(case when version = 'v1' or version = '' then 1 else 0 end) AS hits_v1,
       sum(case when version = 'v2' then 1 else 0 end) AS hits_v2,
       sum(case when (version = 'v1' or version = '') and (date > localtimestamp - interval '1' day) then 1 else 0 end) as hits_v1_1,
       . . .
from logs l
group by api_key;

Answer 2

您可以使用此查询代替您的子查询。

如果你想得到hits

hits给定api-key

您可以选择count(1)获取所有数据。

SELECT api_key,
       count(1) hits,
       SUM(CASE WHEN (version='v1' or version='') THEN 1 ELSE 0 END ) hits_v1,
       SUM(CASE WHEN (version = 'v2' or version='') THEN 1 ELSE 0 END ) hits_v2,
       SUM(CASE WHEN (date > localtimestamp - interval '1' day) THEN 1 ELSE 0 END) hits_1,
       SUM(CASE WHEN (date > localtimestamp - interval '1' day) and (version='v1' or version='') THEN 1 ELSE 0 END) hits_1_v1,
       SUM(CASE WHEN (date > localtimestamp - interval '1' day) and (version='v2' or version='') THEN 1 ELSE 0 END) hits_1_v2,
       SUM(CASE WHEN (date > localtimestamp - interval '7' day) THEN 1 ELSE 0 END) hits_7,
       SUM(CASE WHEN  (version='v1' or version='') and (date > localtimestamp - interval '7' day) THEN 1 ELSE 0 END) hits_7_v1,
       SUM(CASE WHEN  (version='v2' or version='') and (date > localtimestamp - interval '7' day) THEN 1 ELSE 0 END) hits_7_v2
FROM logs
group by api_key

sqlfiddle:http://sqlfiddle.com/#!9/be990/5

多重全外连接（来自同一 table 中的数据）

Multiple full outer join (from data in same table)

sql

join

amazon-athena