HiveQL 查询标记为 table 列名的数据

Question

我在 HDP 2.6.5 平台上使用 Hive (1.2.1000.2.6.5.0-292) 在基于以下数据的简单数据库上工作： https://grouplens.org/datasets/movielens/100k/
我有 4 个 table 命名为：流派、电影、评级、用户，如下所示：

CREATE TABLE genre(genre string, genre_id int);
CREATE TABLE movies (movie_id INT, title STRING, rel_date DATE, video_rel_date STRING, 
imdb_url STRING, unknown INT, action INT, adventure INT, animation INT, childrens INT, 
comedy INT, crime INT, documentary INT, drama INT, fantasy INT, noir INT, horror INT, 
musical INT, mystery INT, romance INT, sci_fi INT, thriller INT, war INT, western INT) 
CLUSTERED BY (movie_id) INTO 12 BUCKETS STORED AS ORC;
CREATE TABLE ratings(user_id int, movie_id int, rating int, rating_time int);
CREATE TABLE users(user_id int, age int, gender char(1), occupation string, zip int);

我想写一个查询 return女性最常看哪种类型的电影，男性最常看哪种类型的电影？但对我来说问题是电影的结构 table 电影类型所在的位置：

1|Toy Story (1995)|1995-01-01||http://us.imdb.com/M/title-exact?Toy%20Story%20(1995)|0|0|0|1|1|1|0|0|0|0|0|0|0|0|0|0|0|0|0

最后 19 个字段是类型，“1”表示电影属于该类型，“0”表示不是。此外，电影可以同时呈现多种类型。性别在 'users' table 中表示为 'M' 或 'F' 字符。所需的 table 可以很容易地加入，但是如何 return 并将列名称的流派分组？

SELECT m.title, r.rating, u.gender
FROM movies m INNER JOIN ratings r ON (m.movie_id = r.movie_id) 
INNER JOIN users u ON (u.user_id = r.user_id);

Answer 1

糟糕的数据模型。您应该有一个 table 每个电影和类型一行。

为了解决这个问题，我建议取消聚合：

select mg.*
from (select m.movie_id, u.gender, count(*) as cnt,
             rank() over (partition by gender order by count(*) desc) as seqnum
      from ((select movie_id, 'action' from movies where action = 1) union all
            (select movie_id, 'adventure' from movies where adventure = 1) union all
            . . .
           ) m join
           ratings r
           on r.movie_id = m.movie_id join
           users u
           on r.user_id = u.user_id
       group by m.movie_id, u.gender
      ) mg
where seqnum = 1;

Answer 2

制作一个流派列数组，按照genre_id对应的顺序排列，分解数组并按位置连接到流派table的数组中。像这样（未测试）：

   select s.title, s.genre, s.gender, s.rating, s.cnt
    from
    (select s.title, s.gender, s.rating, s.cnt, s.genre,
           rank() over (partition by s.gender order by s.cnt desc) as rnk
     from
       (
        select m.title, u.gender, r.rating, g.genre, count(*) over(partition by u.gender) cnt
          from    
            (select m.movie_id, m.title, e.id+1 as genre_id
               from movies m
                  lateral view 
                  posexplode (array(--place columns in a positions corresponding their genre_id
                              unknown, action, adventure, animation, childrens, 
                              comedy, crime, documentary, drama, fantasy, 
                              noir, horror, musical, mystery, romance, 
                              sci_fi, thriller, war, western
                                   )
                             )e as id, val
              where e.val=1
             ) m
             INNER JOIN ratings r ON (m.movie_id = r.movie_id) 
             INNER JOIN users   u ON (u.user_id = r.user_id)
             INNER JOIN genre   g ON (g.genre_id = m.genre_id)
         ) s
    ) s where rnk = 1

HiveQL 查询标记为 table 列名的数据

HiveQL query for data marked as table column names

sql

hive

hiveql

hdp