更改端口 7077 上的绑定 IP 运行 - Apache Spark

Question

能否配置 Spark，而不是绑定到端口 7077 的地址 127.0.1.1，可以而不是绑定到 0.0.0.0 。与绑定端口 8080 的方式相同：

netstat -pln
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.1.1:7077          0.0.0.0:*               LISTEN      2864/java
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      2864/java
tcp        0      0 127.0.1.1:6066          0.0.0.0:*               LISTEN      2864/java
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -
udp        0      0 192.168.192.22:123      0.0.0.0:*                           -
udp        0      0 127.0.0.1:123           0.0.0.0:*                           -
udp        0      0 0.0.0.0:123             0.0.0.0:*                           -
udp        0      0 0.0.0.0:21415           0.0.0.0:*                           -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     7195     -                   /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     SEQPACKET  LISTENING     405      -                   /run/udev/control

我问这个的原因是我无法将工作节点连接到主节点，我认为问题是无法发现主 IP。

尝试将从站连接到主站时出错：

15/04/02 21:58:18 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@raspberrypi:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: raspberrypi/192.168.192.22:7077
15/04/02 21:58:18 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$RegisterWorker] from Actor[akka://sparkWorker/user/Worker#1677101765] to Actor[akka://sparkWorker/deadLetters] was not delivered. [10] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

Answer 1

在spark-env.sh中你可以设置SPARK_MASTER_IP=<ip>.

主机名也可以正常工作（通过 SPARK_STANDALONE_MASTER=<hostname>），只需确保工作人员连接到与主机绑定的主机名完全相同的主机名（即 Spark:// 地址显示在 Spark 主机中UI).

更改端口 7077 上的绑定 IP 运行 - Apache Spark

Change bound IP running on port 7077 - Apache Spark

networking

scala

raspberry-pi

apache-spark