Java: 执行Hadoop MapReduce 有一些问题

Question

我想使用 MapReduce 执行字数统计。我从 hadoop 网站上获取了代码：

package org.myorg;
import java.io.IOException;     
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class WordCount {

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            output.collect(word, one);
       }
     }
   }

public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
            int sum = 0;
            while (values.hasNext()) {
                sum += values.next().get();
            }
            output.collect(key, new IntWritable(sum));
        }
    }   

   public static void main(String[] args) throws Exception {
     JobConf conf = new JobConf(WordCount.class);
     conf.setJobName("wordcount");

     conf.setOutputKeyClass(Text.class);
     conf.setOutputValueClass(IntWritable.class);

     conf.setMapperClass(Map.class);
     conf.setCombinerClass(Reduce.class);
     conf.setReducerClass(Reduce.class);

     conf.setInputFormat(TextInputFormat.class);
     conf.setOutputFormat(TextOutputFormat.class);

     FileInputFormat.setInputPaths(conf, new Path(args[0]));
     FileOutputFormat.setOutputPath(conf, new Path(args[1]));

     JobClient.runJob(conf);
   }
}

我写了这个脚本来编译和执行程序：

#!/bin/bash
#Compile
javac -classpath $HADOOP_INSTALL/share/hadoop/common/hadoop-common-  2.6.5.jar:$HADOOP_INSTALL/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.5.jar:$HADOOP_INSTALL/share/hadoop/common/lib/commons-cli-1.2.jar -d /home/ciro/Scrivania/BDABI/wordcount *.java

#Convert into Jar File
jar -cvf WordCountj.jar -C /home/ciro/Scrivania/BDABI/wordcount/org/myorg .

#Run JAR File
hadoop jar /home/ciro/Scrivania/BDABI/wordcount/WordCountj.jar wordcount /user/inputdata/test.txt outputwc

这些是我收到的错误：

Exception in thread "main" java.lang.ClassNotFoundException: wordcount
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

有什么问题吗？我将单个集群节点（我的电脑）与 ubuntu 14.04 和 hadoop 2.6.5

一起使用

Answer 1

你的 jar 路径和你的 Class 名称在你的作业命令中是错误的

你写了 /home/ciro/Scrivania/BDABI/wordcount/WordCountj.jar 而不是 /home/ciro/Scrivania/BDABI/wordcount/org/myorg/WordCountj.jar 和 wordcount 而不是 WordCount

Answer 2

您的代码与 hadoop website 略有不同；

并且在您的代码中没有工作对象。您必须将 conf 更改为 job 。然后设置路径

Java: 执行Hadoop MapReduce 有一些问题

Java: execution Hadoop MapReduce has some problems

java

apache

hadoop

mapreduce