由于 cloudera quickstart vm 中的纱线容器错误，Hipi 平均像素数程序失败

Question

我是 HIPI/Hadoop 的新手，所以我选择了 cloudera quickstart vm (5.4.2)。我正在按照 getting started 教程进行操作。虽然运行我发现我在 hib 文件中有如下所示的文件：

[cloudera@quickstart tools]$ ./hibInfo.sh examples/sampleNew.hib --show-meta



Input HIB: examples/sampleNew.hib
Display meta data: true
Display EXIF data: false
IMAGE INDEX: 0
   1244 x 829
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/01.jpg}
IMAGE INDEX: 1
   1106 x 829
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/02.jpg}
IMAGE INDEX: 2
   933 x 700
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/03.jpg}
IMAGE INDEX: 3
   1106 x 829
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/04.jpg}
IMAGE INDEX: 4
   1244 x 829
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/05.jpg}
IMAGE INDEX: 5
   1555 x 1166
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/06.jpg}
IMAGE INDEX: 6
   1244 x 829
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/07.jpg}
IMAGE INDEX: 7
   1244 x 829
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/08.jpg}
IMAGE INDEX: 8
   576 x 383
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/09.jpg}
IMAGE INDEX: 9
   576 x 383
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/10.jpg}
IMAGE INDEX: 10
   737 x 475
   format: 1
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/11.jpg}
IMAGE INDEX: 11
   614 x 460
   format: 2
   meta: {source=/home/cloudera/Downloads/hipi-release/web/examples/testimages/12.png}
Found [12] images.

但是当我尝试使用 jar 文件执行时，出现以下错误：

[cloudera@quickstart helloWorld]$ hadoop jar build/libs/helloWorld.jar examples/sampleNew.hib sampleimages_average
15/10/27 03:22:30 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/10/27 03:22:37 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
15/10/27 03:22:40 INFO input.FileInputFormat: Total input paths to process : 1
Spawned 1map tasks
15/10/27 03:22:48 INFO mapreduce.JobSubmitter: number of splits:1
15/10/27 03:22:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1445938202522_0003
15/10/27 03:23:00 INFO impl.YarnClientImpl: Submitted application application_1445938202522_0003
15/10/27 03:23:00 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1445938202522_0003/
15/10/27 03:23:00 INFO mapreduce.Job: Running job: job_1445938202522_0003
15/10/27 03:23:43 INFO mapreduce.Job: Job job_1445938202522_0003 running in uber mode : false
15/10/27 03:23:43 INFO mapreduce.Job:  map 0% reduce 0%
15/10/27 03:23:43 INFO mapreduce.Job: Job job_1445938202522_0003 failed with state FAILED due to: Application application_1445938202522_0003 failed 2 times due to AM Container for appattempt_1445938202522_0003_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1445938202522_0003/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1445938202522_0003_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

资源管理器下的日志文件说：

Failed while trying to construct the redirect url to the log server. Log Server url may not be configured
java.lang.Exception: Unknown container. Container either has not started or has already completed or doesn't belong to this node at all.

帮帮我。我找不到错误。

我正在使用 Java 1.8，Hadoop 2.6.0-cdh5.4.2，

HADOOP_CLASSPATH : /usr/lib/hadoop/lib HADOOP_YARN_HOME : /usr/lib/hadoop-yarn HADOOP_MAPRED_HOME: /usr/lib/hadoop-mapreduce HADOOP_CONF_DIR : /etc/hadoop/conf

我已经用 wordcount 示例进行了测试，结果是成功的。

Answer 1

我通过还原我的 java 版本解决了这个问题，该版本在两个 HIPI 和 hadoop 之间不匹配。我使用 hadoop java 版本构建 hipi，此问题已解决。

由于 cloudera quickstart vm 中的纱线容器错误，Hipi 平均像素数程序失败

Hipi Average no of pixels program fails due to yarn containers error in cloudera quickstart vm

java

hadoop

cloudera-quickstart-vm

hipi