DCOS 集群上的 Spark 提交失败 "Mesos cluster mode is only supported through the REST submission API"

Question

我正在尝试运行在本地 spark-submit，通过 ssh 隧道连接到我的 mesos 主机：

ssh -L 7077:spark-dispatcher.marathon.l4lb.thisdcos.directory:7077 pkara@dcos-master01.mydomain.com

但是我收到以下错误

./spark-submit --deploy-mode cluster --master mesos://localhost:7077 --conf spark.mesos.driverEnv.MYSQL_HOSTNAME= ...

2018-11-14 11:14:18 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.AssertionError: assertion failed: Mesos cluster mode is only supported through the REST submission API
    at scala.Predef$.assert(Predef.scala:170)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:673)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

由于某种原因，当我添加

时问题就消失了

--conf spark.master.rest.enabled=true

DCOS 集群上的 Spark 提交失败 "Mesos cluster mode is only supported through the REST submission API"

Spark submission on DCOS cluster fails with "Mesos cluster mode is only supported through the REST submission API"

mesos

apache-spark

dcos