如何在 Hadoop 2.7.5 中获取用户输入？

Question

我正在努力做到这一点，以便当用户输入一个词时，程序将遍历 txt 文件并计算该词的所有实例。
我正在使用 MapReduce，我是新手。
我知道有一个非常简单的方法可以做到这一点，我已经尝试了一段时间。

在这段代码中，我试图让它请求用户输入，然后程序将遍历文件并找到实例。

我看到一些关于堆栈溢出的代码，有人提到将配置设置为 conf.set("userinput","Data") 会有所帮助。
还有一些更新的方式让用户输入。

我程序中的 if 语句是一个例子，当输入用户词时它只找到那个词。

     import java.util.StringTokenizer;

    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

    public class WordCount {

      public static class TokenizerMapper
           extends Mapper<Object, Text, Text, IntWritable>{

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();


    //So I've seen that this is the correct way of setting it up. 
// However I've heard that there mroe efficeint ways of setting it up as well. 
/*
public void setup(Context context) {
     Configuration config=context.getConfiguration();
     String wordstring=config.get("mapper.word");
     word.setAccessibleHelp(wordstring);
 }
*/


        public void map(Object key, Text value, Context context
                        ) throws IOException, InterruptedException {
          StringTokenizer itr = new StringTokenizer(value.toString());

          while (itr.hasMoreTokens()) {
              if(word=="userinput") {
            word.set(itr.nextToken());
            context.write(word, one);
              }
          }
        }
      }

      public static class IntSumReducer
           extends Reducer<Text,IntWritable,Text,IntWritable> {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable<IntWritable> values,
                           Context context
                           ) throws IOException, InterruptedException {
          int sum = 0;
          for (IntWritable val : values) {
            sum += val.get();
          }
          result.set(sum);
          context.write(key, result);
        }
      }

      public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();


        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
      }
    }

Answer 1

我不确定设置方法，但您将命令行中的输入作为参数传递。

conf.set("mapper.word",args[0]);
Job job =... 
// Notice you now need 3 arguments to run this 
FileInputFormat.addInputPath(job, new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));

在mapper或者reducer中，可以得到字符串

 Configuration config=context.getConfiguration();
 String wordstring=config.get("mapper.word");

并且您需要先从分词器中获取字符串，然后才能进行比较。您还需要比较字符串，而不是将字符串与文本对象进行比较

String wordstring=config.get("mapper.word");
while (itr.hasMoreTokens()) {
    String token = itr.nextToken();
    if(wordstring.equals(token)) {
        word.set(token);
        context.write(word, one);
   }

如何在 Hadoop 2.7.5 中获取用户输入？

How to get user input in Hadoop 2.7.5?

java

hadoop

mapreduce