无法初始化 Cluster Hadoop-2.7.1 和 YARN

Question

对于来自示例的简单程序字数统计提交作业失败，即使所有作业都是运行。

 hadoop jar hadoop-mapreduce-examples-2.7.1.jar wordcount hdfs://localhost:9000/input hdfs://localhost:9000/output

JPS

31265 SecondaryNameNode
31064 DataNode
30929 NameNode
31478 ResourceManager
32354 Jps

错误

java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapreduce.Job.run(Job.java:1260)
    at org.apache.hadoop.mapreduce.Job.run(Job.java:1256)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1255)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1284)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

核心-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/usr/local/hadoop_store/hdfs/tmp</value>
  <description>A base for other temporary directories.</description>
</property>
<property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
</property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
 <configuration>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce.shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name> mapreduce.framework.name </name>
    <value> yarn </value>
   </property>
</configuration>

我尝试了很多其他人的解决方案所以 post 比如改变 mapreduce.shuffle <-> mapreduce_shuffle 但是 none似乎工作。所有目录都存在，网页也可在本地主机上使用。

系统配置：

Ubunutu:15.10
Hadoop: 2.7.1
Java: open-jdk 7u95-2.6.4-0ubuntu0.15.10.1

Answer 1

改变你的 yarn-site.xml 参数

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

Answer 2

从 mapred-site.xml 中删除所有条目后，它起作用了，不知道为什么？

在最新的 2.7.2 版本中 mapred-site.xml.template 已更改为

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
</configuration>

这给了我一个线索，我试过了，它起作用了。

Answer 3

你提到的mapred-site.xml的配置：

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name> mapreduce.framework.name </name>
    <value> yarn </value>
   </property>
</configuration>

此处您已将 MR 作业执行框架配置为 "yarn"，这意味着使用 MRv2。如果您想以传统方式运行您的工作，您可以将框架设置为 "classic" 即 MRv1。你能检查一下这个配置吗？

Answer 4

删除空格：<name> mapreduce.framework.name </name> <value> yarn </value>

然后重试

无法初始化 Cluster Hadoop-2.7.1 和 YARN

Cannot initialize Cluster Hadoop-2.7.1 and YARN

hadoop

hadoop-yarn