Pig 中的连接或过滤器哪个更昂贵？

Question

当在 Pig 中选择加入或过滤时，哪个更耗费性能？

Answer 1

连接总是很昂贵，因为您必须为 table 中的每个元组扫描第二个 table。考虑下面的例子

A = LOAD 'data1' AS (a1:int,a2:int,a3:int);

DUMP A;
(1,2,3)
(4,2,1)
(8,3,4)
(4,3,3)
(7,2,5)
(8,4,3)

B = LOAD 'data2' AS (b1:int,b2:int);

DUMP B;
(2,4)
(8,9)
(1,3)
(2,7)
(2,9)
(4,6)
(4,9)


X = JOIN A BY a1, B BY b1;

DUMP X;
(1,2,3,1,3)
(4,2,1,4,6)
(4,3,3,4,6)
(4,2,1,4,9)
(4,3,3,4,9)
(8,3,4,8,9)
(8,4,3,8,9)

当我们加入X时，我们为A中的每个元组遍历B中的每个元组。对于过滤器，我们只遍历一次数据集并对每个元组执行过滤操作。

X = FILTER A BY a3 == 3;

DUMP X;
(1,2,3)
(4,3,3)
(8,4,3)

Pig 中的连接或过滤器哪个更昂贵？

What is more expensive a join or a filter in Pig?

join

apache-pig