spark submit - 现有连接被远程主机强行关闭 [在主节点上]

Question

我已经在我的 windows 7 本地机器上设置了一个 spark 集群。它有一个主节点和一个工作节点。我使用 sbt 编译 + sbt 包创建了一个简单的 jar，并尝试使用 spark-submit 将它提交到 spark 主节点。目前 master 和 worker 都在同一台机器上，如果它可以工作，那么计划是在多台机器上部署一个本地集群。最终所有这些都将在 Azure 上执行。

主节点

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>spark-class org.apache.spark.deploy.master.Master
2018-12-26 20:00:45 INFO  Master:2612 - Started daemon with process name: 13968@ws-amalhotra
2018-12-26 20:00:45 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-26 20:00:45 INFO  SecurityManager:54 - Changing view acls to: admin
2018-12-26 20:00:45 INFO  SecurityManager:54 - Changing modify acls to: admin
2018-12-26 20:00:45 INFO  SecurityManager:54 - Changing view acls groups to:
2018-12-26 20:00:45 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-12-26 20:00:45 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2018-12-26 20:00:46 INFO  Utils:54 - Successfully started service 'sparkMaster' on port 7077.
2018-12-26 20:00:46 INFO  Master:54 - Starting Spark master at spark://192.168.8.101:7077
2018-12-26 20:00:46 INFO  Master:54 - Running Spark version 2.3.2
2018-12-26 20:00:46 INFO  log:192 - Logging initialized @1268ms
2018-12-26 20:00:46 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-26 20:00:46 INFO  Server:419 - Started @1334ms
2018-12-26 20:00:46 INFO  AbstractConnector:278 - Started ServerConnector@16391414{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2018-12-26 20:00:46 INFO  Utils:54 - Successfully started service 'MasterUI' on port 8080.
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@204e3825{/app,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@748394e8{/app/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@19b99890{/,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5c0f561c{/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3443bda1{/static,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@54541f46{/app/kill,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6e8c3d12{/driver/kill,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  MasterWebUI:54 - Bound MasterWebUI to 0.0.0.0, and started at http://ws-amalhotra.ivp.co.in:8080
2018-12-26 20:00:46 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@22eb9260{/,null,AVAILABLE}
2018-12-26 20:00:46 INFO  AbstractConnector:278 - Started ServerConnector@7e6d0324{HTTP/1.1,[http/1.1]}{192.168.8.101:6066}
2018-12-26 20:00:46 INFO  Server:419 - Started @1394ms
2018-12-26 20:00:46 INFO  Utils:54 - Successfully started service on port 6066.
2018-12-26 20:00:46 INFO  StandaloneRestServer:54 - Started REST server for submitting applications on port 6066
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1a4c3e84{/metrics/master/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5a3b4746{/metrics/applications/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:46 INFO  Master:54 - I have been elected leader! New state: ALIVE
2018-12-26 20:00:54 INFO  Master:54 - Registering worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM
2018-12-26 20:01:20 INFO  Master:54 - Driver submitted org.apache.spark.deploy.worker.DriverWrapper
2018-12-26 20:01:20 INFO  Master:54 - Launching driver driver-20181226200120-0000 on worker worker-20181226200053-192.168.8.101-8089
2018-12-26 20:01:22 INFO  Master:54 - Removing driver: driver-20181226200120-0000
2018-12-26 20:01:25 WARN  TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63501
java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
2018-12-26 20:01:25 WARN  TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63557
java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
2018-12-26 20:01:25 INFO  Master:54 - 192.168.8.101:63501 got disassociated, removing it.
2018-12-26 20:01:25 INFO  Master:54 - 192.168.8.101:63557 got disassociated, removing it.
2018-12-26 20:01:25 INFO  Master:54 - 192.168.8.101:63556 got disassociated, removing it.

工作节点

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Windows\system32>spark-class org.apache.spark.deploy.worker.Worker spark://192.168.8.101:7077 -p 8089
2018-12-26 20:00:53 INFO  Worker:2612 - Started daemon with process name: 13960@ws-amalhotra
2018-12-26 20:00:53 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-12-26 20:00:53 INFO  SecurityManager:54 - Changing view acls to: admin
2018-12-26 20:00:53 INFO  SecurityManager:54 - Changing modify acls to: admin
2018-12-26 20:00:53 INFO  SecurityManager:54 - Changing view acls groups to:
2018-12-26 20:00:53 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-12-26 20:00:53 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); user
s  with modify permissions: Set(admin); groups with modify permissions: Set()
2018-12-26 20:00:53 INFO  Utils:54 - Successfully started service 'sparkWorker' on port 8089.
2018-12-26 20:00:54 INFO  Worker:54 - Starting Spark worker 192.168.8.101:8089 with 8 cores, 14.9 GB RAM
2018-12-26 20:00:54 INFO  Worker:54 - Running Spark version 2.3.2
2018-12-26 20:00:54 INFO  Worker:54 - Spark home: C:\spark
2018-12-26 20:00:54 INFO  log:192 - Logging initialized @1367ms
2018-12-26 20:00:54 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2018-12-26 20:00:54 INFO  Server:419 - Started @1411ms
2018-12-26 20:00:54 INFO  AbstractConnector:278 - Started ServerConnector@319b7858{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2018-12-26 20:00:54 INFO  Utils:54 - Successfully started service 'WorkerUI' on port 8081.
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@32b3c348{/logPage,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3db48501{/logPage/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2606a6f4{/,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6cf2a898{/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@75f7a48a{/static,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@563f35a1{/log,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  WorkerWebUI:54 - Bound WorkerWebUI to 0.0.0.0, and started at http://ws-amalhotra.ivp.co.in:8081
2018-12-26 20:00:54 INFO  Worker:54 - Connecting to master 192.168.8.101:7077...
2018-12-26 20:00:54 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4b2bed3e{/metrics/json,null,AVAILABLE,@Spark}
2018-12-26 20:00:54 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.8.101:7077 after 28 ms (0 ms spent in bootstraps)
2018-12-26 20:00:54 INFO  Worker:54 - Successfully registered with master spark://192.168.8.101:7077
2018-12-26 20:01:20 INFO  Worker:54 - Asked to launch driver driver-20181226200120-0000
2018-12-26 20:01:20 INFO  DriverRunner:54 - Copying user jar file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181226200120-00
00\sbtexample_2.12-0.1.0-SNAPSHOT.jar
2018-12-26 20:01:20 INFO  Utils:54 - Copying D:\_Work\azurepoc\sbtexample\target\scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181226200120-0000\sbtexample_2.12-0.1
.0-SNAPSHOT.jar
2018-12-26 20:01:20 INFO  DriverRunner:54 - Launch Command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.master=spark://19
2.168.8.101:7077" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar"
"-Dspark.rpc.askTimeout=10s" "-Dspark.app.name=example1.HelloWorld" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.8.101:8089" "C:\spark\work\driver-20181226200120-00
00\sbtexample_2.12-0.1.0-SNAPSHOT.jar" "example1.HelloWorld"
2018-12-26 20:01:22 INFO  Worker:54 - Driver driver-20181226200120-0000 exited successfully

Spark-提交代码

C:\Users\amalhotra>spark-submit  --deploy-mode cluster --master spark://192.168.
8.101:7077 --class "example1.HelloWorld"   "D:\_Work\azurepoc\sbtexample\target\
scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"
Running Spark using the REST application submission protocol.
2018-12-26 20:01:09 INFO  RestSubmissionClient:54 - Submitting a request to laun
ch an application in spark://192.168.8.101:7077.
2018-12-26 20:01:19 WARN  RestSubmissionClient:66 - Unable to connect to server
spark://192.168.8.101:7077.
Warning: Master endpoint spark://192.168.8.101:7077 was not a REST server. Falli
ng back to legacy submission gateway instead.
2018-12-26 20:01:19 WARN  NativeCodeLoader:62 - Unable to load native-hadoop lib
rary for your platform... using builtin-java classes where applicable

C:\Users\amalhotra>spark-submit  --deploy-mode cluster --master spark://192.168.
8.101:7077 --class "example1.HelloWorld"   "D:\_Work\azurepoc\sbtexample\target\
scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"

jar码

package example1

import java.io._



object HelloWorld {
    def main(args: Array[String]): Unit = {
         println("===============================================")
         println("===============================================") 
        println("Hello, world!")
        println("===============================================")
        println("===============================================")
    }
}

我在主节点上收到错误：

2018-12-26 20:01:25 WARN  TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63501
java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
2018-12-26 20:01:25 WARN  TransportChannelHandler:78 - Exception in connection from /192.168.8.101:63557
java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)

我保证的事情：

这不是端口相关的问题，因为我已经完全禁用防病毒和防火墙规则。
此外，我已经在我的个人计算机上复制了整个内容（以确保它不是与基础架构相关的问题）
事件查看器中没有内容

编辑：非常感谢 Sc0rpion 的提示。我真是太傻了。我正在尝试将作业提交给 spark master URL。根据 Sc0rpion 共享的 link，它将提交到 REST 端点。我一定是在浏览文档时错过了这一点。

另外，我现在有个小顾虑。我已经通过从我的程序写入文本文件来确保我的工作是运行，但是我无法在主机或工作人员的 控制台上看到打印的任何内容 当我向 spark master 提交作业时。

将作业提交给 master 的代码：

C:\Users\amalhotra>spark-submit  --deploy-mode cluster --master spark://192.168.
8.101:6066 --class "example1.HelloWorld"   "D:\_Work\azurepoc\sbtexample\target\
scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"

师父反应如下：

2018-12-28 13:54:25 INFO  Master:54 - Driver submitted org.apache.spark.deploy.worker.DriverWrapper
2018-12-28 13:54:25 INFO  Master:54 - Launching driver driver-20181228135425-0002 on worker worker-20181228134824-192.168.8.101-8089
2018-12-28 13:54:27 INFO  Master:54 - Removing driver: driver-20181228135425-0002

工人反应：

2018-12-28 13:54:25 INFO  Worker:54 - Asked to launch driver driver-20181228135425-0002
2018-12-28 13:54:25 INFO  DriverRunner:54 - Copying user jar file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181228135425-00
02\sbtexample_2.12-0.1.0-SNAPSHOT.jar
2018-12-28 13:54:25 INFO  Utils:54 - Copying D:\_Work\azurepoc\sbtexample\target\scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar to C:\spark\work\driver-20181228135425-0002\sbtexample_2.12-0.1
.0-SNAPSHOT.jar
2018-12-28 13:54:25 INFO  DriverRunner:54 - Launch Command: "C:\Program Files\Java\jdk1.8.0_181\bin\java" "-cp" "C:\spark\bin\..\conf\;C:\spark\jars\*" "-Xmx1024M" "-Dspark.master=spark://19
2.168.8.101:7077" "-Dspark.driver.supervise=false" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/D:/_Work/azurepoc/sbtexample/target/scala-2.12/sbtexample_2.12-0.1.0-SNAPSHOT.jar"
"-Dspark.app.name=example1.HelloWorld" "org.apache.spark.deploy.worker.DriverWrapper" "spark://Worker@192.168.8.101:8089" "C:\spark\work\driver-20181228135425-0002\sbtexample_2.12-0.1.0-SNAP
SHOT.jar" "example1.HelloWorld"
2018-12-28 13:54:27 INFO  Worker:54 - Driver driver-20181228135425-0002 exited successfully

但是我在任何地方都看不到我的应用程序的输出。它应该像下面这样输出：

Answer 1

正如 link - Unable to submit jobs to spark cluster (cluster-mode)

所指出的

spark-submit --deploy-mode cluster --master spark://192.168.8.101:6066 --class "example1.HelloWorld" "D:_Work\azurepoc\sbtexample\target\scala-2.12\sbtexample_2.12-0.1.0-SNAPSHOT.jar"

而不是端口 7077

spark submit - 现有连接被远程主机强行关闭 [在主节点上]

spark submit - An existing connection was forcibly closed by the remote host [on master node ]

scala

apache-spark

spark-submit

apache-spark-standalone