Pig sum 因 +ve 和 -ve 值而失败

Question

我有以下数据

primary,first,second
1,393440.09,354096.08
1,4410533.33,3969479.99
1,-4803973.41,-4323576.07

我必须对第一列和第二列进行聚合和求和。下面是我正在执行的脚本

   data_load= load <filelocation> using org.apache.pig.piggybank.storage.CSVExcelStorage(',', 'NO_MULTILINE', 'NOCHANGE', 'SKIP_INPUT_HEADER') As  (primary:double, first:double,second:double)

   dataAgrr = group data_load by primary;

   sumData = FOREACH dataAgrr GENERATE 
   group as data,
   SUM(data_load.first) as first,
   SUM(data_load.second) as second,
   SUM(data_load.primary) as primary;

执行后，产生以下输出：

(1.0,0.009999999951105565,-5.820766091346741E-11,3.0)

但是当我们手动添加第二列 (354096.08, 3969479.99, -4323576.07) 时得到 0。

Answer 1

Pig 在内部使用 Java "double"。使用下面的示例代码进行测试

import java.math.BigDecimal;
public class TestSum  {
  public static void main(String[] args) {
    double d1 = 354096.08;
    double d2 = 3969479.99;
    double d3 = -4323576.07;
    System.err.println("Total in double is " + ((d3 + d2 ) + d1));

    BigDecimal bd1 = new BigDecimal("354096.08");
    BigDecimal bd2 = new BigDecimal("3969479.99");
    BigDecimal bd3 = new BigDecimal("-4323576.07");
    System.err.println("Total in BigDecimal is " + bd3.add(bd2).add(bd1));

  }
}

这会产生

Total in double is -5.820766091346741E-11
Total in BigDecimal is 0.00

如果您需要更高的精度，您可能想尝试在脚本中使用 "bigdecimal" 而不是 "double"。

Pig sum 因 +ve 和 -ve 值而失败

Pig sum fails with +ve and -ve values

hadoop

apache-pig