无法在 Apache Tez 上 运行
Unable to run on Apache Tez
感谢您通过屏幕截图详细发布此内容,因为 well.I 已按照您提到的相同步骤进行操作,但我无法安装 properly.The 我遇到的错误是
错误:无法找到或加载主程序 class org.apache.tez.dag.app.DAGAppMaster
我已经下载了tez的构建版本。 Apache Tez Version:0.8.4,Hadoop Version:2.6.0.
我的 tez-site.xml 是
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.8.4,${fs.defaultFS}/apps/tez-0.8.4/lib/</value>
</property>
</configuration>
我的 bashrc 配置是:
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export PATH=$PATH:/usr/local/spark/bin
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:/usr/local/Hadoop/lib/*:.
export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:.
export DERBY_HOME=/usr/local/derby
export PATH=$PATH:$DERBY_HOME/bin
export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytools.jar
export HIVE_OPTS="-hiveconf mapreduce.map.memory.mb=4096 -hiveconf mapreduce.reduce.memory.mb=5120"
export TEZ_HOME=/usr/local/apache-tez-0.8.4-bin
export TEZ_CONF_DIR=$TEZ_HOME/conf
export TEZ_JARS=$TEZ_HOME
if [ -z "$HIVE_AUX_JARS_PATH" ]; then
export HIVE_AUX_JARS_PATH="$TEZ_JARS"
else
export HIVE_AUX_JARS_PATH="$HIVE_AUX_JARS_PATH:$TEZ_JARS"
fi
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
export CLASSPATH=$CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:.
我的 mapreduce-site.xml 是
<configuration>
<!--<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property> -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn-tez</value>
<description>The runtime framework for executing MapReduce jobs.
Can be one of local, classic or yarn.
</description>
</property>
</configuration>
当我尝试 运行 示例示例程序时,它返回跟踪为
Failing this attempt. Failing the application.
16/07/27 12:52:00 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/07/27 12:52:00 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
16/07/27 12:52:00 INFO examples.OrderedWordCount: DAG diagnostics: [Application application_1469604082434_0001 failed 2 times due to AM Container for appattempt_1469604082434_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://AnalyticsLinux.tcs.com:8088/proxy/application_1469604082434_0001/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1469604082434_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
当我在 stderr 下的 http://localhost:8088 中看到时,我找到了上面那个。
请提前帮我解决this.Thanks!!
由于 Tez 是一个孵化器项目,我们需要下载 src 并使用 maven 构建。
Tez 0.8.1 alpha 版本的详细步骤可以参考下面link。
https://acadgild.com/blog/integrating-apache-tez-with-hadoop/
虽然构建它会在 tez-ui.Inorder 上失败,但不要面对这个问题,请在您的 linux 机器上安装 git、节点 js 和 npm,然后开始构建,这将允许你构建成功。
除了上述步骤外,您还需要在~/.bashrc文件中添加参数
export TEZ_CONF_DIR=/usr/local/tez/conf
export TEZ_JARS=/usr/local/tez/tez
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}:${JAVA_JDBC_LIBS}:${MAPREDUCE_LIBS}
export CLASSPATH=$CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
然后尝试 运行 任何成功运行的 mapreduce 作业。
你在 hdfs 中输入了错误的 tez 包。
hdfs dfs -mkdir /application/tez-0.8.4
tar-zxvf apache-tez-0.8.4-bin.tar.gz
hdfs dfs -copyFromLocal apache-tez-0.8.4-bin/share/tez.tar.gz /application/tez-0.8.4
感谢您通过屏幕截图详细发布此内容,因为 well.I 已按照您提到的相同步骤进行操作,但我无法安装 properly.The 我遇到的错误是
错误:无法找到或加载主程序 class org.apache.tez.dag.app.DAGAppMaster
我已经下载了tez的构建版本。 Apache Tez Version:0.8.4,Hadoop Version:2.6.0.
我的 tez-site.xml 是
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.8.4,${fs.defaultFS}/apps/tez-0.8.4/lib/</value>
</property>
</configuration>
我的 bashrc 配置是:
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export PATH=$PATH:/usr/local/spark/bin
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:/usr/local/Hadoop/lib/*:.
export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:.
export DERBY_HOME=/usr/local/derby
export PATH=$PATH:$DERBY_HOME/bin
export CLASSPATH=$CLASSPATH:$DERBY_HOME/lib/derby.jar:$DERBY_HOME/lib/derbytools.jar
export HIVE_OPTS="-hiveconf mapreduce.map.memory.mb=4096 -hiveconf mapreduce.reduce.memory.mb=5120"
export TEZ_HOME=/usr/local/apache-tez-0.8.4-bin
export TEZ_CONF_DIR=$TEZ_HOME/conf
export TEZ_JARS=$TEZ_HOME
if [ -z "$HIVE_AUX_JARS_PATH" ]; then
export HIVE_AUX_JARS_PATH="$TEZ_JARS"
else
export HIVE_AUX_JARS_PATH="$HIVE_AUX_JARS_PATH:$TEZ_JARS"
fi
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
export CLASSPATH=$CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:.
我的 mapreduce-site.xml 是
<configuration>
<!--<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property> -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn-tez</value>
<description>The runtime framework for executing MapReduce jobs.
Can be one of local, classic or yarn.
</description>
</property>
</configuration>
当我尝试 运行 示例示例程序时,它返回跟踪为
Failing this attempt. Failing the application.
16/07/27 12:52:00 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/07/27 12:52:00 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
16/07/27 12:52:00 INFO examples.OrderedWordCount: DAG diagnostics: [Application application_1469604082434_0001 failed 2 times due to AM Container for appattempt_1469604082434_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://AnalyticsLinux.tcs.com:8088/proxy/application_1469604082434_0001/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1469604082434_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
当我在 stderr 下的 http://localhost:8088 中看到时,我找到了上面那个。
请提前帮我解决this.Thanks!!
由于 Tez 是一个孵化器项目,我们需要下载 src 并使用 maven 构建。
Tez 0.8.1 alpha 版本的详细步骤可以参考下面link。
https://acadgild.com/blog/integrating-apache-tez-with-hadoop/
虽然构建它会在 tez-ui.Inorder 上失败,但不要面对这个问题,请在您的 linux 机器上安装 git、节点 js 和 npm,然后开始构建,这将允许你构建成功。
除了上述步骤外,您还需要在~/.bashrc文件中添加参数
export TEZ_CONF_DIR=/usr/local/tez/conf
export TEZ_JARS=/usr/local/tez/tez
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}:${JAVA_JDBC_LIBS}:${MAPREDUCE_LIBS}
export CLASSPATH=$CLASSPATH:${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*
然后尝试 运行 任何成功运行的 mapreduce 作业。
你在 hdfs 中输入了错误的 tez 包。
hdfs dfs -mkdir /application/tez-0.8.4
tar-zxvf apache-tez-0.8.4-bin.tar.gz
hdfs dfs -copyFromLocal apache-tez-0.8.4-bin/share/tez.tar.gz /application/tez-0.8.4