Nutch/Hadoop: regex-normalize.xml 和 regex-urlfilter.txt 未找到错误，即使它们存在

Question

我正在尝试运行通过 eclipse 进行 nutch 和 hadoop 并按照几个教程进行设置。我目前陷入了一个 nullpointerexception，我认为这是由于未找到 regex-urlfilter.txt 和 regex-normalize.xml 引起的。

这是来自日志的错误跟踪：-

[LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.conf.Configuration  - regex-normalize.xml not found
4473 [LocalJobRunner Map Task Executor #0] WARN org.apache.nutch.net.urlnormalizer.regex.RegexURLNormalizer  - Can't load the default rules! 
4477 [LocalJobRunner Map Task Executor #0] DEBUG org.apache.nutch.util.ObjectCache  - No object cache found for conf=Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, file:/tmp/hadoop-338737067/mapred/local/localRunner/338737067/job_local1524701719_0001/job_local1524701719_0001.xml, instantiating a new object cache
4486 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.conf.Configuration  - regex-urlfilter.txt not found
4486 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask  - Starting flush of map output
4516 [LocalJobRunner Map Task Executor #0] DEBUG org.apache.hadoop.util.concurrent.ExecutorHelper  - afterExecute in thread: LocalJobRunner Map Task Executor #0, runnable type: java.util.concurrent.FutureTask
4516 [Thread-3] INFO org.apache.hadoop.mapred.LocalJobRunner  - map task executor complete.
4521 [Thread-3] WARN org.apache.hadoop.mapred.LocalJobRunner  - job_local1524701719_0001
java.lang.Exception: java.lang.NullPointerException
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:491)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:551)

这两个文件都存在于 \workspace\apache-nutch-1.16\conf 文件夹中，我不确定我做错了什么。我仔细检查了我的环境变量是否为 HADOOP_HOME 和 HADOOP_BIN 设置正确，并且它们指向正确的目录。我不确定他们正在寻找哪个目录来查找 regex-urlfilter.txt 和 regex-normalize.xml。任何解决此问题的帮助将不胜感激。

我正在使用 Hadoop 3.0.0 和 apache-nutch-1.16。

Answer 1

conf/ 文件夹需要位于 Java 类路径中。运行 Nutch 使用提供的脚本之一 bin/nutch 或 bin/crawl 最容易做到这一点。如果使用二进制包，则脚本位置为apache-nutch-1.16/bin/nutch。对于源码包，它是 apache-nutch-1.16/runtime/local/bin/nutch 在 ant runtime 被执行之后。使用脚本还允许将配置文件放在不同的目录中，并将 NUTCH_CONF_DIR 指向该目录。脚本只会将此位置放在类路径的前面。

Nutch/Hadoop: regex-normalize.xml 和 regex-urlfilter.txt 未找到错误，即使它们存在

Nutch/Hadoop: regex-normalize.xml and regex-urlfilter.txt not found error even though they exist

eclipse

hadoop

nutch