无法在代码中将应用程序提交到 spark-cluster

Question

我的代码是：

SparkSession spark = SparkSession.builder().appName("plzzzz").master("local[*]").getOrCreate();
Dataset<Row> dataset = spark.read().format("libsvm").load("/home/ceny/tools/123.txt");
KMeans kmeans = new KMeans().setK(2).setSeed(1L);
KMeansModel model = kmeans.fit(dataset);//this line is 20

错误是：

16/12/19 00:08:49 WARN KMeans: The input data was not directly cached, which may hurt performance if its parent RDDs are also uncached.
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
    at org.apache.spark.ml.clustering.KMeansModel.transform(KMeans.scala:124)
    at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:326)
    at Test.main(Test.java:20)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)

我的build.gradle是：

compile group: 'org.scala-lang', name: 'scala-reflect', version: '2.11.8'
compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.0.2'
compile group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.0.2'
compile group: 'org.apache.spark', name: 'spark-mllib_2.10', version: '2.0.2'

我的环境是：

spark-version:2.0.2

scala-version:2.11.8

描述：

我尝试在IDEA中执行下面的代码，出现错误。如果我把它打包成JAR，然后在终端提交，一切都很好。

而如果我使用SparkSubmit.main()通过代码提交jar，仍然出现错误。

我现在该怎么办？

Answer 1

所有工件需要相同的 Scala 版本。应该是：

compile group: 'org.apache.spark', name: 'spark-mllib_2.11', version: '2.0.2'

无法在代码中将应用程序提交到 spark-cluster

Can not submit application to spark-cluster in code

java

scala

apache-spark

apache-spark-mllib