使用 MapReduce 解析文件 Java

Question

我正在尝试使用 Hadoop MapReduce 解析一个 json 文件，但在编译时出现了我在之前的 MapReduce 项目中没有遇到的奇怪错误。

Mapper.java:43: error: type Mapper does not take parameters
public static class Map extends Mapper<LongWritable, Text, Text, Text> {
                                      ^
Mapper.java:45: error: cannot find symbol
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
                                                  ^
  symbol:   class Context
  location: class Map
Mapper.java:35: error: incompatible types: Class<Map> cannot be converted to Class<? extends Mapper>
    job.setMapperClass(Map.class);

我的代码如下：

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.*;
import org.apache.hadoop.mapreduce.lib.output.*;
import org.apache.hadoop.util.*;

public class Mapper extends Configured implements Tool {

public static void main(String args[]) throws Exception {
    int res = ToolRunner.run(new Mapper(), args);
    System.exit(res);
}

public int run(String[] args) throws Exception {
    Path inputPath = new Path(args[0]);
    Path outputPath = new Path(args[1]);

    Configuration conf = getConf();
    Job job = new Job(conf, this.getClass().toString());

    FileInputFormat.setInputPaths(job, inputPath);
    FileOutputFormat.setOutputPath(job, outputPath);

    job.setJobName("Mapper");
    job.setJarByClass(Mapper.class);

    job.setMapperClass(Map.class);

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    return job.waitForCompletion(true) ? 0 : 1;
}

public static class Map extends Mapper<LongWritable, Text, Text, Text> {

    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        String line = value.toString();
        Link link = Link.parse(line);
        context.write(new Text(link.url().toString()), new Text(link.tags().toString()));
    }
}
}

如果能帮助我确定收到这些错误的原因，我们将不胜感激。谢谢。

Answer 1

您的主要 class 称为 Mapper，它又嵌套了 class，扩展了 Mapper。将您的 main class 重命名为其他名称。还要将 Map 更改为其他内容，否则可能会与 java.util.Map.

混淆

使用 MapReduce 解析文件 Java

Parsing File with MapReduce Java

java

hadoop

mapreduce