如何使用 spark-submit 和 python 脚本访问 Docker 容器中的 spark 集群？

Question

我创建了一个 Spark 集群，其中有一个主服务器和两个从服务器，每个都在一个 Docker 容器上。我使用命令 start-all.sh.

启动它

我可以从位于 localhost:8080 的本地计算机访问 UI，它表明集群已正常启动： Screenshot of Spark UI

然后我尝试使用此命令 spark-submit 从我的主机（而不是 Docker 容器）提交一个简单的 Python 脚本：spark-submit --master spark://spark-master:7077 test.py

test.py :

import pyspark
conf = pyspark.SparkConf().setAppName('MyApp').setMaster('spark://spark-master:7077')
sc = pyspark.SparkContext(conf=conf)

但是控制台返回了这个错误：

22/01/26 09:20:39 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://spark-master:7077...
22/01/26 09:20:40 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master spark-master:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
    at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$$anon.run(StandaloneAppClient.scala:106)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Failed to connect to spark-master:7077
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
    at org.apache.spark.rpc.netty.Outbox$$anon.call(Outbox.scala:194)
    at org.apache.spark.rpc.netty.Outbox$$anon.call(Outbox.scala:190)
    ... 4 more
Caused by: java.net.UnknownHostException: spark-master
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
    at java.base/java.net.InetAddress.getByName(InetAddress.java:1252)
    at io.netty.util.internal.SocketUtils.run(SocketUtils.java:146)
    at io.netty.util.internal.SocketUtils.run(SocketUtils.java:143)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
    at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
    at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
    at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)
    at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108)
    at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:202)
    at io.netty.bootstrap.Bootstrap.access[=13=]0(Bootstrap.java:48)
    at io.netty.bootstrap.Bootstrap.operationComplete(Bootstrap.java:182)
    at io.netty.bootstrap.Bootstrap.operationComplete(Bootstrap.java:168)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
    at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:985)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:505)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.access0(AbstractChannel.java:416)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.run(AbstractChannel.java:475)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:518)
    at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    ... 1 more

我也尝试使用一个简单的 scala 脚本，只是为了尝试访问集群，但我遇到了同样的错误。

您知道如何使用 python 脚本访问我的集群吗？

（编辑）

我忘了说明我已经在我的主人和我的奴隶之间创建了一个 docker 网络。因此，在 MrTshoot 和 Gaarv 的帮助下，我将 spark-master（在 spark://spark-master:7077 中）替换为我的主容器的 ip（您可以使用命令 docker network inspect my-network 获取它）。

成功了！谢谢！

Answer 1

当您指定 .setMaster('spark://spark-master:7077') 时，它意味着“在本地计算机无法解析的 DNS 地址“spark-master”和端口 7077 到达 spark 集群。

因此为了让您的主机到达集群，您必须改为指定您的 Spark 集群的 Docker DNS / IP 地址，检查本地计算机上的“docker0”接口并替换“spark-master" 用它。

Answer 2

您无法使用其服务名称 连接主机上的Docker 服务。您应该设置 DNS 或 IP 服务或使用以下技巧：

公开您的 Spark 集群端口。
打开你的 /etc/hosts 并在上面添加以下内容

127.0.0.1 本地主机 spark-master

::1 本地主机 spark-master

如何使用 spark-submit 和 python 脚本访问 Docker 容器中的 spark 集群？

How can I reach a spark cluster in a Docker container with spark-submit and a python script?

python

docker

apache-spark

pyspark

spark-submit