Spark 2.0 Scala 导入语句

Question

在 Spark 1.6.2 (Scala 2.10.5) 上，以下代码在 shell:

中运行良好

import org.apache.spark.mllib.linalg.Vector
case class DataPoint(vid: String, label: Double, features: Vector)

mllib Vector 正确地掩盖了 Scala Vector。

但是，在 Spark 2.0 (Scala 2.11.8) 上，相同的代码会在 shell 中抛出以下错误：

<console>:11: error: type Vector takes type parameters
  case class DataPoint(vid: String, label: Double, features: Vector)

为了让它工作，我现在必须明确命名 class：

case class DataPoint(vid: String, label: Double,
  features: org.apache.spark.mllib.linalg.Vector)

有人可以告诉我发生了什么变化吗？Spark 或 Scala 有问题吗？谢谢！

Answer 1

这个问题最简单的解决方案是一个简单的 paste:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0-SNAPSHOT
      /_/

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_102)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.linalg.Vector

scala> case class DataPoint(vid: String, label: Double, features: Vector)
<console>:11: error: type Vector takes type parameters
       case class DataPoint(vid: String, label: Double, features: Vector)
                                                                  ^

scala> :paste
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.mllib.linalg.Vector
case class DataPoint(vid: String, label: Double, features: Vector)

// Exiting paste mode, now interpreting.

import org.apache.spark.mllib.linalg.Vector
defined class DataPoint

Spark 2.0 Scala 导入语句

Spark 2.0 Scala import statements

scala

apache-spark

apache-spark-mllib