java 中 Spark DataFrame.flatMap 的使用

Question

我在 java 中使用过 RDD.flatMap 函数。现在尝试使用 DataFrames。

他们说：

public <R> RDD<R> flatMap(scala.Function1<org.apache.spark.sql.Row,
    scala.collection.TraversableOnce<R>> f, scala.reflect.ClassTag<R> evidence)

Returns a new RDD by first applying a function to all rows of this DataFrame, and then flattening the results.

Specified by: flatMap in interface RDDApi

但是当我尝试这个时，Function1 迫使我重写很多很多未实现的方法。这是我得到的：

    RDD<Row> res = df.flatMap(new Function1<Row, TraversableOnce<Row>>() {

        @Override
        public <A> Function1<Row, A> andThen(
                Function1<TraversableOnce<Row>, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcDD$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcDF$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcDI$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcDJ$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcFD$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcFF$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcFI$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcFJ$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcID$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcIF$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcII$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcIJ$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcJD$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcJF$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcJI$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcJJ$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcVD$sp(
                Function1<BoxedUnit, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcVF$sp(
                Function1<BoxedUnit, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcVI$sp(
                Function1<BoxedUnit, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcVJ$sp(
                Function1<BoxedUnit, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcZD$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcZF$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcZI$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<Object, A> andThen$mcZJ$sp(
                Function1<Object, A> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public TraversableOnce<Row> apply(Row arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public double apply$mcDD$sp(double arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public double apply$mcDF$sp(float arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public double apply$mcDI$sp(int arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public double apply$mcDJ$sp(long arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public float apply$mcFD$sp(double arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public float apply$mcFF$sp(float arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public float apply$mcFI$sp(int arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public float apply$mcFJ$sp(long arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public int apply$mcID$sp(double arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public int apply$mcIF$sp(float arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public int apply$mcII$sp(int arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public int apply$mcIJ$sp(long arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public long apply$mcJD$sp(double arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public long apply$mcJF$sp(float arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public long apply$mcJI$sp(int arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public long apply$mcJJ$sp(long arg0) {
            // TODO Auto-generated method stub
            return 0;
        }

        @Override
        public void apply$mcVD$sp(double arg0) {
            // TODO Auto-generated method stub

        }

        @Override
        public void apply$mcVF$sp(float arg0) {
            // TODO Auto-generated method stub

        }

        @Override
        public void apply$mcVI$sp(int arg0) {
            // TODO Auto-generated method stub

        }

        @Override
        public void apply$mcVJ$sp(long arg0) {
            // TODO Auto-generated method stub

        }

        @Override
        public boolean apply$mcZD$sp(double arg0) {
            // TODO Auto-generated method stub
            return false;
        }

        @Override
        public boolean apply$mcZF$sp(float arg0) {
            // TODO Auto-generated method stub
            return false;
        }

        @Override
        public boolean apply$mcZI$sp(int arg0) {
            // TODO Auto-generated method stub
            return false;
        }

        @Override
        public boolean apply$mcZJ$sp(long arg0) {
            // TODO Auto-generated method stub
            return false;
        }

        @Override
        public <A> Function1<A, TraversableOnce<Row>> compose(
                Function1<A, Row> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcDD$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcDF$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcDI$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcDJ$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcFD$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcFF$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcFI$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcFJ$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcID$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcIF$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcII$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcIJ$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcJD$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcJF$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcJI$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcJJ$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, BoxedUnit> compose$mcVD$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, BoxedUnit> compose$mcVF$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, BoxedUnit> compose$mcVI$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, BoxedUnit> compose$mcVJ$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcZD$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcZF$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcZI$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }

        @Override
        public <A> Function1<A, Object> compose$mcZJ$sp(
                Function1<A, Object> arg0) {
            // TODO Auto-generated method stub
            return null;
        }
    }, evidence);

这看起来很奇怪，但我继续将 evidence 设为：

ClassTag<Row> evidence = scala.reflect.ClassTag$.MODULE$.apply(Row.class);

我的意图是玩玩 flatMap（当然是在 DataFrames 上，而不是在 RDD 上）。所以我不需要对 Row 进行任何更改。可以return输入原样，不做任何更改。

所以我想我只需要在 apply 方法中进行更改。

    @Override
    public TraversableOnce<Row> apply(Row arg0) {
        // TODO Auto-generated method stub
        return null;
    }

但是，我应该如何从 Row 得到 TraversableOnce<Row>？

另外，我尝试的方法是否正确？还是我遗漏了什么？

我正在使用 Apache Spark 1.3.1

Answer 1

您应该执行如下操作：

RDD<Row> res = df.flatMap(new AbstractFunction1<Row, TraversableOnce<Row>>() {
  public TraversableOnce<Row> apply(Row row) {
    return new ListSet<Row>().$plus(row); //Note the updated list is returned from $plus()
  }
}, evidence);

这与 map 的工作方式类似，只是可以更自由地更改。例如，要过滤掉某些内容，您可以 return 空的 new ListSet<Row>() 当您想要 return 时，或者保持当前行为。 flatMap 非常灵活。

（似乎从 Java 集合到 Scala 集合的转换并不简单。）

java 中 Spark DataFrame.flatMap 的使用

Usage of Spark DataFrame.flatMap in java

java

flatmap

apache-spark