运行 Spark 应用程序：java.lang.NoSuchMethod 错误
Running Spark application: java.lang.NoSuchMethod error
我有一个使用 sbt 0.13.7、Scala 2.11.4 和 Spark 1.2.0 组装的 Spark 应用程序。我正在 运行宁 Mac OSX Yosemite。
在 build.sbt 中，我使用 "provided" 作为 Spark 依赖项。我可以 运行 sbt 中的应用程序。
当我尝试从命令行 运行 它时，我 运行 遇到了问题。这是我使用的命令：
ADD_JARS=analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar scala -cp /Applications/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar com.dtex.analysis.transform.GenUserSummaryView
我在下面收到以下错误消息。请告知我可以做些什么来解决这个问题。谢谢！
一个运行
15/01/12 22:47:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/01/12 22:47:18 WARN BlockManager: Putting block broadcast_0 failed
java.lang.NoSuchMethodError: scala.collection.immutable.$colon$colon.hd()Ljava/lang/Object;
at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:84)
at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
at org.apache.spark.util.collection.SizeTrackingVector.resetSamples(SizeTrackingVector.scala:25)
at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
at org.apache.spark.util.collection.SizeTrackingVector.<init>(SizeTrackingVector.scala:25)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:236)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:136)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:114)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:787)
at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:638)
at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:992)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:98)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:945)
at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:695)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:540)
at com.dtex.analysis.transform.TransformUtils$anonfun.apply(TransformUtils.scala:97)
at com.dtex.analysis.transform.TransformUtils$anonfun.apply(TransformUtils.scala:97)
at scala.collection.TraversableLike$anonfun$map.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$anonfun$map.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at com.dtex.analysis.transform.TransformUtils$.generateUserSummaryData(TransformUtils.scala:97)
at com.dtex.analysis.transform.GenUserSummaryView$.main(GenUserSummaryView.scala:77)
at com.dtex.analysis.transform.GenUserSummaryView.main(GenUserSummaryView.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at scala.reflect.internal.util.ScalaClassLoader$anonfun$run.apply(ScalaClassLoader.scala:70)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:101)
at scala.reflect.internal.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:70)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:22)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:39)
at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:29)
at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:39)
at scala.tools.nsc.MainGenericRunner.runTarget(MainGenericRunner.scala:65)
at scala.tools.nsc.MainGenericRunner.run(MainGenericRunner.scala:87)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Spark 1.2.0 是针对 Scala 2.10 构建的，因此您需要针对 2.10 版本的 Scala 构建您的程序。
我有一个使用 sbt 0.13.7、Scala 2.11.4 和 Spark 1.2.0 组装的 Spark 应用程序。我正在 运行宁 Mac OSX Yosemite。
在 build.sbt 中，我使用 "provided" 作为 Spark 依赖项。我可以 运行 sbt 中的应用程序。
当我尝试从命令行 运行 它时，我 运行 遇到了问题。这是我使用的命令：
ADD_JARS=analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar scala -cp /Applications/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar:analysis/target/scala-2.11/dtex-analysis_2.11-0.1.jar com.dtex.analysis.transform.GenUserSummaryView
我在下面收到以下错误消息。请告知我可以做些什么来解决这个问题。谢谢！
一个运行
15/01/12 22:47:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/01/12 22:47:18 WARN BlockManager: Putting block broadcast_0 failed
java.lang.NoSuchMethodError: scala.collection.immutable.$colon$colon.hd()Ljava/lang/Object;
at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:84)
at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
at org.apache.spark.util.collection.SizeTrackingVector.resetSamples(SizeTrackingVector.scala:25)
at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
at org.apache.spark.util.collection.SizeTrackingVector.<init>(SizeTrackingVector.scala:25)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:236)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:136)
at org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:114)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:787)
at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:638)
at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:992)
at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:98)
at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
at org.apache.spark.SparkContext.broadcast(SparkContext.scala:945)
at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:695)
at org.apache.spark.SparkContext.textFile(SparkContext.scala:540)
at com.dtex.analysis.transform.TransformUtils$anonfun.apply(TransformUtils.scala:97)
at com.dtex.analysis.transform.TransformUtils$anonfun.apply(TransformUtils.scala:97)
at scala.collection.TraversableLike$anonfun$map.apply(TraversableLike.scala:245)
at scala.collection.TraversableLike$anonfun$map.apply(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at com.dtex.analysis.transform.TransformUtils$.generateUserSummaryData(TransformUtils.scala:97)
at com.dtex.analysis.transform.GenUserSummaryView$.main(GenUserSummaryView.scala:77)
at com.dtex.analysis.transform.GenUserSummaryView.main(GenUserSummaryView.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at scala.reflect.internal.util.ScalaClassLoader$anonfun$run.apply(ScalaClassLoader.scala:70)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:101)
at scala.reflect.internal.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:70)
at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:22)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:39)
at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:29)
at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:39)
at scala.tools.nsc.MainGenericRunner.runTarget(MainGenericRunner.scala:65)
at scala.tools.nsc.MainGenericRunner.run(MainGenericRunner.scala:87)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Spark 1.2.0 是针对 Scala 2.10 构建的，因此您需要针对 2.10 版本的 Scala 构建您的程序。