运行 Mapreduce 问题

Question

我正在尝试运行集群 hadoop 2.7.1（一个主服务器和 4 个从服务器）上的一个 wordcount jar，但是 MapReduce 作业被阻止在：

$ hadoop jar wc.jar WordCount /input /output_hocine 
17/03/13 09:41:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
17/03/13 09:41:43 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 
17/03/13 09:41:43 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 
17/03/13 09:41:44 INFO input.FileInputFormat: Total input paths to process : 3 
17/03/13 09:41:44 INFO mapreduce.JobSubmitter: number of splits:3 
17/03/13 09:41:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1489393376058_0003 
17/03/13 09:41:44 INFO impl.YarnClientImpl: Submitted application application_1489393376058_0003 
17/03/13 09:41:44 INFO mapreduce.Job: The url to track the job: http://ibnbadis21:8088/proxy/application_1489393376058_0003/ 
17/03/13 09:41:44 INFO mapreduce.Job: Running job: job_1489393376058_0003

通过导航器，此图显示了通过导航器的输出：

配置文件的内容如下：

核心-site.xml:

<configuration>
<!--    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://ibnbadis21:9000</value>
    </property>-->
    <property>
        <name>fs.default.name</name>
        <value>hdfs://ibnbadis21:9000</value>
    </property>
   <property>
      <name>dfs.permissions</name>
      <value>false</value>
   </property>
 </configuration>

纱-site.xml:

<?xml version="1.0"?> <configuration>
   <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

mapred-site.xml:

<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
    http://www.apache.org/licenses/LICENSE-2.0
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. -->
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.address</name>
                <value>ibnbadis21:10020</value>
        </property>
        <property>
                <name>mapreduce.jobhistory.webapp.address</name>
                <value>ibnbadis21:19888</value>
        </property>
        <property>
                <name>yarn.app.mapreduce.am.staging-dir</name>
                <value>/user/app</value>
        </property>
</configuration>

hdfs-site.xml:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
 <property>
        <name>dfs.replication</name>
        <value>2</value>
 </property>
 <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/local/hadoop/hadoop_data/hdfs/namenode</value>
 </property>
 <property> <name>dfs.namenode.checkpoint.dir</name>
        <value>file:/usr/local/hadoop_data/hdfs/namesecondary</value>
 </property>
 <property> <name>dfs.datanode.data.dir</name>
        <value>file:/usr/local/hadoop_data/hdfs/datanode</value>
 </property>
</configuration>

谁能告诉我如何解决这个问题？

Answer 1

Connecting to ResourceManager at /0.0.0.0:8032

0.0.0.0（默认值）不是有效的主机名。

因此，将此添加到 yarn-site.xml

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value> YOUR VALUE HERE </value> <!-- Needs Fully Qualified Domain Name -->
</property>

您可能没有设置很多值。

参考Hadoop | Configuring the Hadoop Daemons

顺便说一下，fs.defaultFS 是正确的属性。

Answer 2

最后是访问权限的问题。该框架无权访问我的 yarn-site.xml 文件。这就是它使用默认值 0.0.0.0/8030 的原因。因此，当我以特权 (sudo) 执行命令时：

sudo hadoop jar wc.jar WordCount /input /output

我的作业MapReduce执行成功！

运行 Mapreduce 问题

Running Mapreduce issues

hadoop

mapreduce

hadoop-yarn