使用 Apache Spark 2.0.2 启动 spark-jobserver 时出错

Question

我正在尝试启动 spark-jobserver。我在 GitHub spark_jobserver page 上提供的安装步骤中找不到任何对此 akka 库的引用。我是运行在一台服务器上以独立模式运行的 spark，它充当 master/node.

但是当我执行

./job-server/server_start.sh

它显示以下错误：

error while starting up loggers
akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.slf4j.Slf4jLogger] due to [java.lang.ClassNotFoundException: akka.event.slf4j.Slf4jLogger]
    at akka.event.LoggingBus$$anonfun$$anonfun$apply.applyOrElse(Logging.scala:116)
    at akka.event.LoggingBus$$anonfun$$anonfun$apply.applyOrElse(Logging.scala:115)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at scala.util.Failure$$anonfun$recover.apply(Try.scala:216)
    at scala.util.Try$.apply(Try.scala:192)
    at scala.util.Failure.recover(Try.scala:216)
    at akka.event.LoggingBus$$anonfun.apply(Logging.scala:115)
    at akka.event.LoggingBus$$anonfun.apply(Logging.scala:110)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map.apply(TraversableLike.scala:683)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682)
    at akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:110)
    at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
    at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:623)
    at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:157)
    at akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
    at akka.actor.ActorSystemImpl.liftedTree2(ActorSystem.scala:620)
    at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
    at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
    at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
    at spark.jobserver.JobServer$.spark$jobserver$JobServer$$makeSupervisorSystem(JobServer.scala:154)
    at spark.jobserver.JobServer$$anonfun$main.apply(JobServer.scala:156)
    at spark.jobserver.JobServer$$anonfun$main.apply(JobServer.scala:156)
    at spark.jobserver.JobServer$.start(JobServer.scala:54)
    at spark.jobserver.JobServer$.main(JobServer.scala:156)
    at spark.jobserver.JobServer.main(JobServer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: akka.event.slf4j.Slf4jLogger
    at java.net.URLClassLoader.run(URLClassLoader.java:366)
    at java.net.URLClassLoader.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:274)
    at akka.actor.ReflectiveDynamicAccess$$anonfun$getClassFor.apply(DynamicAccess.scala:67)
    at akka.actor.ReflectiveDynamicAccess$$anonfun$getClassFor.apply(DynamicAccess.scala:66)
    at scala.util.Try$.apply(Try.scala:192)
    at akka.actor.ReflectiveDynamicAccess.getClassFor(DynamicAccess.scala:66)
    at akka.event.LoggingBus$$anonfun.apply(Logging.scala:113)
    ... 33 more
Exception in thread "main" akka.ConfigurationException: Could not start logger due to [akka.ConfigurationException: Logger specified in config can't be loaded [akka.event.slf4j.Slf4jLogger] due to [java.lang.ClassNotFoundException: akka.event.slf4j.Slf4jLogger]]
    at akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:144)
    at akka.event.EventStream.startDefaultLoggers(EventStream.scala:26)
    at akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:623)
    at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:157)
    at akka.cluster.ClusterActorRefProvider.init(ClusterActorRefProvider.scala:58)
    at akka.actor.ActorSystemImpl.liftedTree2(ActorSystem.scala:620)
    at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:617)
    at akka.actor.ActorSystemImpl._start(ActorSystem.scala:617)
    at akka.actor.ActorSystemImpl.start(ActorSystem.scala:634)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
    at spark.jobserver.JobServer$.spark$jobserver$JobServer$$makeSupervisorSystem(JobServer.scala:154)
    at spark.jobserver.JobServer$$anonfun$main.apply(JobServer.scala:156)
    at spark.jobserver.JobServer$$anonfun$main.apply(JobServer.scala:156)
    at spark.jobserver.JobServer$.start(JobServer.scala:54)
    at spark.jobserver.JobServer$.main(JobServer.scala:156)
    at spark.jobserver.JobServer.main(JobServer.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

解决这个问题的两个步骤：

执行后创建的spark-job-server.jar ./bin/server_package.sh 它必须移动到$SPARK_HOME[=16=上的jars文件夹]
尽管如此，我还是不得不下载 akka-slf4j_2.11.0-RC3-2.3.0.jar 并将其也放在 $SPARK_HOME/jars 文件夹中。

使用 Apache Spark 2.0.2 启动 spark-jobserver 时出错

Error starting spark-jobserver with Apache Spark 2.0.2

akka

apache-spark

spark-jobserver