apache spark predictProbabilities 不是成员 NaiveBayesModel

Question

当我执行 sbt package 或 sbt assembly 时，我得到如下错误：

值 predictProbabilities 不是 org.apache.spark.mllib.classification.NaiveBayesModel

的成员

但相同的代码在我的 spark-shell

中有效

/home/user1/spark-1.5.2/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.4.0

我使用 spark-1.5.2 和 scala 2.10.4 sbt 0.13.8

我用的是centos 6.5 linux

我确实在这个 link 中看到了 predictProbabilities。 https://spark.apache.org/docs/1.5.2/api/scala/index.html#org.apache.spark.mllib.classification.NaiveBayesModel

我不知道是否需要 sbt assembly 或 io.prediction 但我试过了。没什么区别。得到同样的错误。

有没有人遇到过这样的问题。如果是这样，你能告诉我解决方案吗？

谢谢！

我的密码是

============================================= =============

import org.apache.spark.mllib.classification.{NaiveBayes, NaiveBayesModel}

import org.apache.spark.mllib.util.MLUtils

val trainData = MLUtils.loadLibSVMFile(sc, "file:///home/user1/nbtest/trainData.nb.svm")

val testData = MLUtils.loadLibSVMFile(sc, "file:///home/user1/nbtest/testData.nb.svm")

val model = NaiveBayes.train(trainData, lambda = 1.0, modelType = "multinomial")

val predictionAndLabel = testData.map(p => (model.predict(p.features), p.label))

val accuracy = 1.0 * predictionAndLabel.filter(x => x._1 == x._2).count() / testData.count()

val testDataFeatures = testData.map(p => p.features)

val probabilities = model.predictProbabilities (testDataFeatures)

============================================= ======================

我的/home/user1/nbtest/build.sbt是这样的

============================================= ================

name := "testBayes"

version := "1.0" 

scalaVersion := "2.10.4"

scalaSource in Compile := baseDirectory.value / ""

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2"

libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.5.2"

libraryDependencies += "org.apache.spark"  % "spark-mllib_2.10" % "1.4.0"

libraryDependencies += "com.databricks"  % "spark-csv_2.10" % "1.4.0"

libraryDependencies += "io.prediction" % "core_2.10" % "0.9.6"

libraryDependencies += "io.prediction" % "sdk" % "0.5.0"   

assemblyMergeStrategy in assembly := {
 case PathList("META-INF", xs @ _*) => MergeStrategy.discard
 case x => MergeStrategy.first

============================================= ================

我有 /home/user1/nbtest/project assembly.sbt plugins.sbt

assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")

plugins.sbt

logLevel := Level.Warn

============================================= ================

Answer 1

使用此库代替您正在使用的 spark-mllib 1.4.0 :

libraryDependencies += "org.apache.spark"  % "spark-mllib_2.10" % "1.5.2"

apache spark predictProbabilities 不是成员 NaiveBayesModel

apache spark predictProbabilities not a member NaiveBayesModel

apache-spark

apache-spark-mllib