java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils at SimpleApp.main(SimpleApp.java:61)

Question

为了项目要求，我试图构建 FlumUtils 示例，其中 spark 示例中存在 spark。我能够创建 jar 文件。但是在尝试执行它时出现以下错误。谁能帮我解决这个问题？

Error: application failed with exception
java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils
        at SimpleApp.main(SimpleApp.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.flume.FlumeUtils
        at java.net.URLClassLoader.run(URLClassLoader.java:366)
        at java.net.URLClassLoader.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)




  I have included the below dependency as pom file



<dependency> <!-- Spark Flume dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-flume_2.10</artifactId>
  <version>1.2.1</version>
</dependency>
<dependency> <!-- Spark Core dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.2.1</version>
</dependency>
<dependency> <!-- Spark Steaming dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.10</artifactId>
  <version>1.2.1</version>

而我运行与 spark-submit 相同 --class SimpleApp target/simple-project-1.0.jar

谁能帮我做同样的事情？

Answer 1

如果您运行使用 spark-submit 的作业.. 请确保您使用 提供 spark-streaming_2.10 jar --jars 选项

Answer 2

正在从 https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-flume-assembly 下载 spark-streaming-flume-assembly_2.10.jar，并且运行使用选项 --jars spark-streaming-flume-assembly_2.10.jar 进行 spark-submit，应该解决NoClassDefFound错误的问题，这是由于运行时依赖解析问题引起的。

java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils at SimpleApp.main(SimpleApp.java:61)

java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils at SimpleApp.main(SimpleApp.java:61)

maven

flume

apache-spark