如何以编程方式在 yarn-client 模式下提交 spark 应用程序？

Question

我有一个简单的 spark 作业，它在给定的输入文件中用逗号替换空格。

当这个作业在本地提交时（使用 IDE 并执行构建的 jar）它成功完成并且当 master 设置为 "yarn-client" 作业挂起很长时间并抛出以下内容异常。

我们有一个用例，我们希望以编程方式提交作业，而不是构建 jar 并通过 spark-submit 提交。

星火版本：1.6.1 Hadoop 版本：2.7.1

并且我在我的 pom 中获得了所有的 spark、yarn 和 hadoop 依赖项。

作业因以下异常而失败

java.net.ConnectException: Call From spark.node123.com/192.168.2.1 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
    at org.apache.hadoop.ipc.Client.call(Client.java:1480)
    at org.apache.hadoop.ipc.Client.call(Client.java:1407)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy10.getClusterMetrics(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:152)
    at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy11.getClusterMetrics(Unknown Source)
    at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:246)
    at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication.apply(Client.scala:129)
    at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication.apply(Client.scala:129)
    at org.apache.spark.Logging$class.logInfo(Logging.scala:58)
    at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:62)
    at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:128)
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:530)
    at tardis.platform.TardisContext$.apply(TardisContext.scala:20)
    at tardis.common.plugins.Heartbeat.isAbleTocreateContext(Heartbeat.scala:45)
    at tardis.common.plugins.Heartbeat.performAction(Heartbeat.scala:33)
    at tardis.core.scheduler.jobs.PluginExecutorJob.execute(PluginExecutorJob.scala:40)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707)
    at org.apache.hadoop.ipc.Client$Connection.access00(Client.java:370)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529)
    at org.apache.hadoop.ipc.Client.call(Client.java:1446)
    ... 25 more

Answer 1

您无法在客户端模式下远程提交您的 Spark 作业，因为您的计算机必须运行驱动程序本身需要大量连接。如果您坚持使用此方法，则必须配置防火墙以允许某些端口连接到集群。使用集群模式或从主节点提交它要容易得多。

Answer 2

我必须添加 hadoop 和 yarn 配置才能在 yarn-client 模式下成功提交应用程序。

如何以编程方式在 yarn-client 模式下提交 spark 应用程序？

How to programatically submit a spark application in yarn-client mode?

scala

hadoop-yarn

apache-spark