无法找到或加载主 class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
mapred-site.xml 的内容:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
虽然我设置了yarn.app.mapreduce.am.env等参数;我收到 Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster 错误。
我正在尝试 运行 远程映射 reduce 程序,其中 hadoop 安装在 linux 机器上,我正在 运行 从 windows 机器上安装它。
以下是我的工作配置设置。
public class WordCount {
public static void main(String[] args)
throws IOException, ClassNotFoundException, InterruptedException {
//
UserGroupInformation ugi = UserGroupInformation.createRemoteUser("admin");
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
try {
Configuration configuration = new Configuration();
configuration.set("yarn.resourcemanager.address", "192.168.33.75:50001"); // see step 3
configuration.set("mapreduce.framework.name", "yarn");
// configuration.set("yarn.app.mapreduce.am.env",
// "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
// configuration.set("mapreduce.map.env", "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
// configuration.set("mapreduce.reduce.env",
// "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
configuration.set("fs.defaultFS", "hdfs://192.168.33.75:54310"); // see step 2
configuration.set("mapreduce.app-submission.cross-platform", "true");
configuration.set("mapred.remote.os", "Linux");
configuration.set("yarn.application.classpath",
"$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:"
+ "$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:"
+ "$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:"
+ "$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*");
Job job = Job.getInstance(configuration);
job.setJarByClass(WordCount.class); // use this when uploaded the Jar to the server and
// running the job directly and locally on the server
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(MapForWordCount.class);
job.setReducerClass(ReduceForWordCount.class);
Path input = new Path("/user/admin/wordCountInput.txt");
Path output = new Path("/user/admin/output");
FileInputFormat.addInputPath(job, input);
FileOutputFormat.setOutputPath(job, output);
System.exit(job.waitForCompletion(true) ? 0 : 1);
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
});
}
请帮助我。从过去的 6 天开始,我就被这个问题困住了。非常感谢。
Hadoop 版本:3.1.0
我遇到了同样的问题,通过在 mapred-site.xml 中添加这个解决了(所以编辑你的 mapreduce.application.classpath 属性)
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/common/*,$HADOOP_MAPRED_HOME/share/hadoop/common/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value>
</property>
我在 yarn-site.xml
中添加了以下 属性
<property>
<name>yarn.application.classpath</name>
<value> $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
</value>
</property>
并在我的 map reduce 程序中进行了相同的更改。
configuration.set("yarn.application.classpath",
"{{HADOOP_CONF_DIR}},{{HADOOP_COMMON_HOME}}/share/hadoop/common/*,{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*,"
+ " {{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*,{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*,"
+ "{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*,"
+ "{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*,{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*");
我的程序现在运行很顺利。欢迎询问我详情。
问题是您的资源管理器 (yarn) 无法加载 Hadoop 库 (jars)。
我通过更新配置解决了这个问题。
将此添加到 yarn-site.xml :
<property>
<name>yarn.application.classpath</name>
<value>C:/hadoop-2.8.0/share/hadoop/mapreduce/*,C:/hadoop-2.8.0/share/hadoop/mapreduce/lib/*,C:/Hadoop-2.8.0/share/hadoop/common/*,C:/Hadoop-2.8.0/share/hadoop/common/lib/*,
C:/hadoop-2.8.0/share/hadoop/hdfs/*,C:/hadoop-2.8.0/share/hadoop/hdfs/lib/*,C:/hadoop-2.8.0/share/hadoop/yarn/*,C:/hadoop-2.8.0/share/hadoop/yarn/lib/*</value>
</property>
请注意,根据您的系统,此处使用的路径可以是相对路径。
只需编辑 mapred-site.xml 文件即可:
添加以下属性:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/Users/adityaatri/Applications/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/Users/adityaatri/Applications/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/Users/adityaatri/Applications/hadoop-3.1.3</value>
</property>
现在,对于上述所有 4 个属性,将路径 /Users/adityaatri/Applications/hadoop-3.1.3
替换为您的 Hadoop 主地址。
现在添加第 5 个 属性 :
<property>
<name>mapreduce.application.classpath</name>
<value></value>
</property>
在终端中执行以下命令后,<value>
元素必须填充内容:
export HADOOP_CLASSPATH=$(hadoop classpath)
echo $HADOOP_CLASSPATH
我终端的输出:
/Users/adityaatri/Applications/hadoop-3.1.3/etc/hadoop:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/common/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/common/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/hdfs:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/hdfs/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/mapreduce/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/yarn:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/yarn/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/yarn/*
将此内容复制到第 5 个 属性 的 <value>
元素中。
现在您将不会收到任何错误。 :)
我在 mapred.xml
中添加了 mapreduce.application.classpath
,但它仍然不起作用。然后我从我的linux服务器下载了core-site.xml
、mapred-site.xml
和yarn-site.xml
文件,然后把这些文件放到你IDEA的resources
路径下,就可以了!
mapred-site.xml 的内容:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>
虽然我设置了yarn.app.mapreduce.am.env等参数;我收到 Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster 错误。 我正在尝试 运行 远程映射 reduce 程序,其中 hadoop 安装在 linux 机器上,我正在 运行 从 windows 机器上安装它。 以下是我的工作配置设置。
public class WordCount {
public static void main(String[] args)
throws IOException, ClassNotFoundException, InterruptedException {
//
UserGroupInformation ugi = UserGroupInformation.createRemoteUser("admin");
ugi.doAs(new PrivilegedExceptionAction<Void>() {
public Void run() throws Exception {
try {
Configuration configuration = new Configuration();
configuration.set("yarn.resourcemanager.address", "192.168.33.75:50001"); // see step 3
configuration.set("mapreduce.framework.name", "yarn");
// configuration.set("yarn.app.mapreduce.am.env",
// "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
// configuration.set("mapreduce.map.env", "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
// configuration.set("mapreduce.reduce.env",
// "HADOOP_MAPRED_HOME=/home/admin/hadoop-3.1.0");
configuration.set("fs.defaultFS", "hdfs://192.168.33.75:54310"); // see step 2
configuration.set("mapreduce.app-submission.cross-platform", "true");
configuration.set("mapred.remote.os", "Linux");
configuration.set("yarn.application.classpath",
"$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/*:$HADOOP_COMMON_HOME/lib/*:"
+ "$HADOOP_HDFS_HOME/*:$HADOOP_HDFS_HOME/lib/*:"
+ "$HADOOP_YARN_HOME/*:$HADOOP_YARN_HOME/lib/*:"
+ "$HADOOP_MAPRED_HOME/*:$HADOOP_MAPRED_HOME/lib/*");
Job job = Job.getInstance(configuration);
job.setJarByClass(WordCount.class); // use this when uploaded the Jar to the server and
// running the job directly and locally on the server
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(MapForWordCount.class);
job.setReducerClass(ReduceForWordCount.class);
Path input = new Path("/user/admin/wordCountInput.txt");
Path output = new Path("/user/admin/output");
FileInputFormat.addInputPath(job, input);
FileOutputFormat.setOutputPath(job, output);
System.exit(job.waitForCompletion(true) ? 0 : 1);
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
});
}
请帮助我。从过去的 6 天开始,我就被这个问题困住了。非常感谢。 Hadoop 版本:3.1.0
我遇到了同样的问题,通过在 mapred-site.xml 中添加这个解决了(所以编辑你的 mapreduce.application.classpath 属性)
<property>
<name>mapreduce.application.classpath</name>
<value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/common/*,$HADOOP_MAPRED_HOME/share/hadoop/common/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/*,$HADOOP_MAPRED_HOME/share/hadoop/yarn/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/*,$HADOOP_MAPRED_HOME/share/hadoop/hdfs/lib/*</value>
</property>
我在 yarn-site.xml
中添加了以下 属性<property>
<name>yarn.application.classpath</name>
<value> $HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,
$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
</value>
</property>
并在我的 map reduce 程序中进行了相同的更改。
configuration.set("yarn.application.classpath",
"{{HADOOP_CONF_DIR}},{{HADOOP_COMMON_HOME}}/share/hadoop/common/*,{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*,"
+ " {{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*,{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*,"
+ "{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/*,{{HADOOP_MAPRED_HOME}}/share/hadoop/mapreduce/lib/*,"
+ "{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*,{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*");
我的程序现在运行很顺利。欢迎询问我详情。
问题是您的资源管理器 (yarn) 无法加载 Hadoop 库 (jars)。 我通过更新配置解决了这个问题。 将此添加到 yarn-site.xml :
<property> <name>yarn.application.classpath</name> <value>C:/hadoop-2.8.0/share/hadoop/mapreduce/*,C:/hadoop-2.8.0/share/hadoop/mapreduce/lib/*,C:/Hadoop-2.8.0/share/hadoop/common/*,C:/Hadoop-2.8.0/share/hadoop/common/lib/*, C:/hadoop-2.8.0/share/hadoop/hdfs/*,C:/hadoop-2.8.0/share/hadoop/hdfs/lib/*,C:/hadoop-2.8.0/share/hadoop/yarn/*,C:/hadoop-2.8.0/share/hadoop/yarn/lib/*</value> </property>
请注意,根据您的系统,此处使用的路径可以是相对路径。
只需编辑 mapred-site.xml 文件即可:
添加以下属性:
<property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
<property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=/Users/adityaatri/Applications/hadoop-3.1.3</value> </property>
<property> <name>mapreduce.map.env</name> <value>HADOOP_MAPRED_HOME=/Users/adityaatri/Applications/hadoop-3.1.3</value> </property>
<property> <name>mapreduce.reduce.env</name> <value>HADOOP_MAPRED_HOME=/Users/adityaatri/Applications/hadoop-3.1.3</value> </property>
现在,对于上述所有 4 个属性,将路径 /Users/adityaatri/Applications/hadoop-3.1.3
替换为您的 Hadoop 主地址。
现在添加第 5 个 属性 :
<property> <name>mapreduce.application.classpath</name> <value></value> </property>
在终端中执行以下命令后,<value>
元素必须填充内容:
export HADOOP_CLASSPATH=$(hadoop classpath)
echo $HADOOP_CLASSPATH
我终端的输出:
/Users/adityaatri/Applications/hadoop-3.1.3/etc/hadoop:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/common/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/common/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/hdfs:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/hdfs/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/mapreduce/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/yarn:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/yarn/lib/*:/Users/adityaatri/Applications/hadoop-3.1.3/share/hadoop/yarn/*
将此内容复制到第 5 个 属性 的 <value>
元素中。
现在您将不会收到任何错误。 :)
我在 mapred.xml
中添加了 mapreduce.application.classpath
,但它仍然不起作用。然后我从我的linux服务器下载了core-site.xml
、mapred-site.xml
和yarn-site.xml
文件,然后把这些文件放到你IDEA的resources
路径下,就可以了!