java.lang.NoClassDefFoundError: better/files/File in spark-sbumit for scala code
java.lang.NoClassDefFoundError: better/files/File in spark-sbumit for scala code
当我将程序导出到 jar 文件并执行它时,我收到 java.lang.NoClassDefFoundError: better/files/File 错误。
我使用的代码如下。
在此先感谢您的帮助
SBT
name := "testFunctions"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "com.github.pathikrit" %% "better-files" % "2.17.1"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.1.0",
"org.apache.spark" % "spark-sql_2.11" % "2.1.0",
"com.github.pathikrit" %% "better-files" % "2.17.1"
)
initialize := {
val _ = initialize.value
if (sys.props("java.specification.version") != "1.8")
sys.error("Java 8 is required for this project.")
}
Scala 代码
/**
* Created by cloudera on 7/23/17.
*/
import better.files.File._
import org.apache.spark.sql.SparkSession
object funcJM {
val forDelete = (root/"/home/cloudera/Documents/fabo")
.createIfNotExists()
if (forDelete.exists)
forDelete.delete()
def main(args:Array[String]) : Unit = {
val spark = SparkSession.builder
.master("local")
.appName("Get ForEx Data")
.getOrCreate()
}
}
命令行执行jar文件
spark-submit --class funcJM --master local[*] /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar --driver-class-path /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar
错误
Exception in thread "main" java.lang.NoClassDefFoundError:
better/files/File at java.lang.Class.getDeclaredMethods0(Native
Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at
java.lang.Class.getMethod0(Class.java:3018) at
java.lang.Class.getMethod(Class.java:1784) at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:722)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused
by: java.lang.ClassNotFoundException: better.files.File at
java.net.URLClassLoader.findClass(URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 10 more
代码所在的树结构如下所示 -
所以 class 调用的 func 存在于 scala 目录中
\main
\java
\resources
\scala
-funcJM(class)
\scala-2.11
您可以做三件事:
使用sbt assembly之类的东西来编译一个fat jar。包含 better.files.File 的 jar 将被打包在 testfunctions_2.11-1.0.jar.
将包含更好包的jar文件复制到spark jars目录中:../spark-2.1.0-bin-hadoop2.7/jars
使用 --driver-class-path better-files-akka_2.10.jar
在您的 spark-submit 调用中指定包含更好包的 jar 文件作为参数
当我将程序导出到 jar 文件并执行它时,我收到 java.lang.NoClassDefFoundError: better/files/File 错误。
我使用的代码如下。
在此先感谢您的帮助
SBT
name := "testFunctions"
version := "1.0"
scalaVersion := "2.11.8"
libraryDependencies += "com.github.pathikrit" %% "better-files" % "2.17.1"
libraryDependencies ++= Seq(
"org.apache.spark" % "spark-core_2.11" % "2.1.0",
"org.apache.spark" % "spark-sql_2.11" % "2.1.0",
"com.github.pathikrit" %% "better-files" % "2.17.1"
)
initialize := {
val _ = initialize.value
if (sys.props("java.specification.version") != "1.8")
sys.error("Java 8 is required for this project.")
}
Scala 代码
/**
* Created by cloudera on 7/23/17.
*/
import better.files.File._
import org.apache.spark.sql.SparkSession
object funcJM {
val forDelete = (root/"/home/cloudera/Documents/fabo")
.createIfNotExists()
if (forDelete.exists)
forDelete.delete()
def main(args:Array[String]) : Unit = {
val spark = SparkSession.builder
.master("local")
.appName("Get ForEx Data")
.getOrCreate()
}
}
命令行执行jar文件
spark-submit --class funcJM --master local[*] /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar --driver-class-path /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar
错误
Exception in thread "main" java.lang.NoClassDefFoundError: better/files/File at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:722) at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: better.files.File at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 10 more
代码所在的树结构如下所示 - 所以 class 调用的 func 存在于 scala 目录中
\main
\java
\resources
\scala
-funcJM(class)
\scala-2.11
您可以做三件事:
使用sbt assembly之类的东西来编译一个fat jar。包含 better.files.File 的 jar 将被打包在 testfunctions_2.11-1.0.jar.
将包含更好包的jar文件复制到spark jars目录中:../spark-2.1.0-bin-hadoop2.7/jars
使用
--driver-class-path better-files-akka_2.10.jar
在您的 spark-submit 调用中指定包含更好包的 jar 文件作为参数