在 YARN 上设置 Spark 作业服务器

Question

我正在尝试在我的 YARN 上设置 spark 作业服务器我在做

./bin/server_package.sh ec2

这会在 /tmp/jobserver 中创建一个文件夹当我尝试 ./sever_start.sh

我收到这个错误

Uncaught error from thread [JobServer-akka.actor.default-dispatcher-2] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
Uncaught error from thread [JobServer-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: ooyala/common/akka/ActorMetrics$$anonfun$receive
        at ooyala.common.akka.ActorMetrics$class.receive(ActorMetrics.scala:20)
        at ooyala.common.akka.InstrumentedActor.receive(InstrumentedActor.scala:8)
        at akka.actor.ActorCell.newActor(ActorCell.scala:558)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        at akka.actor.ActorCell.invokeAll(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Uncaught error from thread [JobServer-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[JobServer]
java.lang.NoClassDefFoundError: ooyala/common/akka/ActorMetrics$$anonfun$receive
        at ooyala.common.akka.ActorMetrics$class.receive(ActorMetrics.scala:20)
        at ooyala.common.akka.InstrumentedActor.receive(InstrumentedActor.scala:8)
        at akka.actor.ActorCell.newActor(ActorCell.scala:558)
        at akka.actor.ActorCell.create(ActorCell.scala:578)
        at akka.actor.ActorCell.invokeAll(ActorCell.scala:456)
        at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
        at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
java.lang.NoClassDefFoundError: scala/runtime/AbstractPartialFunction$mcVL$sp
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

试过减少版本的akka没有效果有什么建议吗？

Answer 1

您使用的 sbt 版本是否与构建 spark jobserver 的版本相同？

Answer 2

问题似乎出在所使用的 scala 版本冲突中。 Spark 在 2.11.7 而作业服务器在 2.10.4

Answer 3

始终尝试匹配 Spark 和 spark job-server 的 scala 版本。 ec2.sh/local.sh 中的版本应与您在 spark-shell 中看到的 scala 版本相同。

在 YARN 上设置 Spark 作业服务器

Setting up Spark jobserver on YARN

hadoop-yarn

apache-spark

spark-jobserver