"Bad substitution" 将 spark 作业提交到 yarn-cluster 时

Question

我正在使用 yarn-cluster 作为 master 和 SparkPi 示例程序对 yarn 集群进行冒烟测试。这是命令行：

  $SPARK_HOME/bin/spark-submit --master yarn-cluster 
 --executor-memory 8G --executor-cores 240 --class org.apache.spark.examples.SparkPi

examples/target/scala-2.11/spark-examples-1.4.1-hadoop2.7.1.jar

纱线接受了工作，但随后抱怨 "bad substitution"。也许它在 hdp.version ??

15/09/01 21:54:05 INFO yarn.Client: Application report for application_1441066518301_0013 (state: ACCEPTED)
15/09/01 21:54:05 INFO yarn.Client:
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1441144443866
     final status: UNDEFINED
     tracking URL: http://yarnmaster-8245.lvs01.dev.ebayc3.com:8088/proxy/application_1441066518301_0013/
     user: stack
15/09/01 21:54:06 INFO yarn.Client: Application report for application_1441066518301_0013 (state: ACCEPTED)
15/09/01 21:54:10 INFO yarn.Client: Application report for application_1441066518301_0013 (state: FAILED)
15/09/01 21:54:10 INFO yarn.Client:
     client token: N/A
     diagnostics: Application application_1441066518301_0013 failed 2 times due to AM Container for appattempt_1441066518301_0013_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://yarnmaster-8245.lvs01.dev.ebayc3.com:8088/cluster/app/application_1441066518301_0013Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e03_1441066518301_0013_02_000001
Exit code: 1
Exception message: /mnt/yarn/nm/local/usercache/stack/appcache/
application_1441066518301_0013/container_e03_1441066518301_0013_02_000001/
launch_container.sh: line 24: $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:
/usr/hdp/current/hadoop-client/*::$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-.6.0.${hdp.version}.jar:
/etc/hadoop/conf/secure: bad substitution

Stack trace: ExitCodeException exitCode=1: /mnt/yarn/nm/local/usercache/stack/appcache/application_1441066518301_0013/container_e03_1441066518301_0013_02_000001/launch_container.sh: line 24: $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution

    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
    at org.apache.hadoop.util.Shell.run(Shell.java:456)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

这里值得注意的是：

/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-.6.0.${hdp.version}.jar:
/etc/hadoop/conf/secure: bad substitution

"sh" 链接到 bash:

$ ll /bin/sh
lrwxrwxrwx 1 root root 4 Sep  1 05:48 /bin/sh -> bash

Answer 1

这可能是由于 /bin/sh 链接到破折号，而不是 bash，这通常发生在基于 Debian 的系统上。

要修复它，运行 sudo dpkg-reconfigure dash 和 select 没有。

Answer 2

这是由于hdp.version没有正确替换造成的。您必须在 $SPARK_HOME/conf 下的文件 java-opts 中设置 hdp.version。

而且你必须设置

spark.driver.extraJavaOptions -Dhdp.version=XXX 
spark.yarn.am.extraJavaOptions -Dhdp.version=XXX

in spark-defaults.conf under $SPARK_HOME/conf 其中 XXX 是 hdp 的版本。

Answer 3

我遇到了同样的问题：

launch_container.sh: line 24: $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*::$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution

因为我找不到任何 /usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo* 文件，我只是编辑了 mapred-site.xml 并删除了 "/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:"

Answer 4

如果您将 spark 与 hdp 一起使用，则必须执行以下操作：

在 $SPARK_HOME/conf/spark-defaults.conf

中添加这些条目

spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)

spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)

在 $SPARK_HOME/conf 中创建一个名为 java-opts 的文件，并将安装的 HDP 版本添加到该文件中，如下所示：

-Dhdp.version=2.2.0.0-2041 (your installed HDP version)

要确定安装的是哪个hdp版本，请在集群中运行这个命令：

hdp-select status hadoop-client

Answer 5

我在使用带有 yarn、spark 和 mapreduce 2 的 BigInsights 4.2.0.0 时也遇到了这个问题，导致它的原因是 iop.version。要修复它，您必须将 iop.version 变量添加到 mapred-site，这可以通过以下步骤完成：

在 Ambari 服务器中转到：

MAPREDUCE2
配置（选项卡）
高级（选项卡）
点击进入自定义 mapred-site
添加属性...
输入 iop.version 和您的 BigInsights 版本。
重新启动所有服务。

这已经解决了。

Answer 6

转到 ambari-yarn。

点击配置->高级->自定义纱线站点->添加属性 ...

添加 hdp 版本作为键和值作为您的 HDP 版本。您将获得 hdp 版本以及以下命令

hdp-select 版本

例如2.5.3.0-37

现在加你属性为

hdp.version=2.5.3.0-37

否则将 yarn-site.xml 和 yarn-env.sh

"Bad substitution" 将 spark 作业提交到 yarn-cluster 时

"Bad substitution" when submitting spark job to yarn-cluster

hadoop-yarn

apache-spark