将 worker 添加到 Spark Standalone CDH5.3

Question

我在 VM 上安装了 cloudera cdh5.3 quickstart 运行ning。我在使用运行ning Spark 时遇到问题。我已经完成了这些步骤 http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_spark_configure... 和运行这个词 exapmle 并且它起作用了。但是当我去 master (quickstart.cloudera:18080) 那里没有工人 cores=0, memory=0... 当我去 (quickstart.cloudera:18081) 那里有一个工人。我的问题是如何添加工人？我应该在 export STANDALONE_SPARK_MASTER_HOST?

中输入什么

这是火花-env.sh:

#Change the following to specify a real cluster's Master host
export STANDALONE_SPARK_MASTER_HOST=worker-20150402201049-10.0.2.15-7078
export SPARK_MASTER_IP=$STANDALONE_SPARK_MASTER_HOST
### Let's run everything with JVM runtime, instead of Scala
export SPARK_LAUNCH_WITH_SCALA=0
export SPARK_LIBRARY_PATH=${SPARK_HOME}/lib
export SCALA_LIBRARY_PATH=${SPARK_HOME}/lib
export SPARK_MASTER_WEBUI_PORT=18080
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_PORT=7078
export SPARK_WORKER_WEBUI_PORT=18081
export SPARK_WORKER_DIR=/var/run/spark/work
export SPARK_LOG_DIR=/var/log/spark
export SPARK_PID_DIR='/var/run/spark/'
if [ -n "$HADOOP_HOME" ]; then
export LD_LIBRARY_PATH=:/lib/native
fi
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-etc/hadoop/conf}
### Comment above 2 lines and uncomment the following if
### you want to run with scala version, that is included with the package
#export SCALA_HOME=${SCALA_HOME:-/usr/lib/spark/scala}
#export PATH=$PATH:$SCALA_HOME/bin

谢谢

Answer 1

将 export STANDALONE_SPARK_MASTER_HOST=10.0.2.15 添加到您的 spark-env.sh 以便主节点和工作节点同意相同的主机地址。

将 worker 添加到 Spark Standalone CDH5.3

Adding worker to Spark Standalone CDH5.3

cloudera

apache-spark

cloudera-cdh

cloudera-quickstart-vm