Hadoop mapreduce 自定义可写静态上下文

Question

我正在做大学作业，我们必须为此使用 hadoop mapreduce。我正在尝试创建一个新的自定义可写对象，因为我想将键值对输出为 (key, (doc_name, 1))。

public class Detector {

    private static final Path TEMP_PATH = new Path("temp");
    private static final String LENGTH = "gramLength";
    private static final String THRESHOLD = "threshold";


    public class Custom implements Writable {

        private Text document;
        private IntWritable count;

        public Custom(){
            setDocument("");
            setCount(0);
        }

        public Custom(String document, int count) {
            setDocument(document);
            setCount(count);
        }

        @Override
        public void readFields(DataInput in) throws IOException {
            // TODO Auto-generated method stub
            document.readFields(in);
            count.readFields(in);
        }

        @Override
        public void write(DataOutput out) throws IOException {
            document.write(out);
            count.write(out);
        }

        public int getCount() {
            return count.get();
        }

        public void setCount(int count) {
            this.count = new IntWritable(count);
        }

        public String getDocument() {
            return document.toString();
        }

        public void setDocument(String document) {
            this.document = new Text(document);
        }

    }

    public static class NGramMapper extends Mapper<Text, Text, Text, Text> {
        private int gramLength;
        private Pattern space_pattern=Pattern.compile("[ ]");
        private StringBuilder gramBuilder= new StringBuilder();

        @Override
        protected void setup(Context context) throws IOException,      InterruptedException{
            gramLength=context.getConfiguration().getInt(LENGTH, 0);
        }

        public void map(Text key, Text value, Context context) throws IOException, InterruptedException {
            String[] tokens=space_pattern.split(value.toString());
            for(int i=0;i<tokens.length;i++){
                gramBuilder.setLength(0);
                if(i+gramLength<=tokens.length){
                    for(int j=i;j<i+gramLength;j++){
                        gramBuilder.append(tokens[j]);
                        gramBuilder.append(" ");
                    }
                    context.write(new Text(gramBuilder.toString()), key);
                }
            }
        }
    }


    public static class OutputReducer extends Reducer<Text, Text, Text, Custom> {

        public void reduce(Text key, Iterable<Text> values, Context context)
                throws IOException, InterruptedException {
            for (Text val : values) {
                context.write(key,new Custom(val.toString(),1));
            }
        }
    }

    public static void main(String[] args) throws Exception {

        Configuration conf = new Configuration();
        FileSystem fs = FileSystem.get(conf);
        conf.setInt(LENGTH, Integer.parseInt(args[0]));
        conf.setInt(THRESHOLD, Integer.parseInt(args[1]));

        // Setup first MapReduce phase
        Job job1 = Job.getInstance(conf, "WordOrder-first");
        job1.setJarByClass(Detector.class);
        job1.setMapperClass(NGramMapper.class);
        job1.setReducerClass(OutputReducer.class);
        job1.setMapOutputKeyClass(Text.class);
        job1.setMapOutputValueClass(Text.class);
        job1.setOutputKeyClass(Text.class);
        job1.setOutputValueClass(Custom.class);
        job1.setInputFormatClass(WholeFileInputFormat.class);
        FileInputFormat.addInputPath(job1, new Path(args[2]));
        FileOutputFormat.setOutputPath(job1, new Path(args[3]));

        boolean status1 = job1.waitForCompletion(true);
        if (!status1) {
            System.exit(1);
        }
    }
}

当我将代码编译为 class 文件时，出现此错误：

Detector.java:147: error: non-static variable this cannot be referenced from a static context
context.write(key,new Custom(val.toString(),1));

我遵循了关于自定义可写的不同教程，我的解决方案与其他人相同。有什么建议吗？

Answer 1

静态字段和方法与所有实例共享。它们用于特定于 class 而不是特定实例的值。尽量远离他们。

要解决你的问题，你需要实例化你的class的一个实例（创建一个对象），这样run-time可以为实例预留内存；或将您正在访问的部分更改为具有 static 访问权限（不推荐！）。

关键字 this 用于引用确实是实例的东西（因此 this 东西）而不是 static 的东西，在那种情况下应该由 class 名称引用。您在不允许的 static 上下文中使用它。

Hadoop mapreduce 自定义可写静态上下文

Hadoop mapreduce custom writable static context

java

hadoop

mapreduce

writable