容器应要求终止，而运行映射作业

Question

我是运行 benchmark tearasort 程序，mapper 阶段进展顺利。但是当谈到减速器阶段时，我得到以下信息。最终工作失败了

16/05/20 14:42:53 INFO mapreduce.Job: Task Id : attempt_1463557283514_0017_r_000008_1, Status : FAILED

Container [pid=54068,containerID=container_1463557283514_0017_01_018425] is running beyond physical memory limits. Current usage: 4.2 GB of 4 GB physical memory used; 8.1 GB of 8.4 GB virtual memory used. Killing container.

Dump of the process-tree for container_1463557283514_0017_01_018425 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 54068 54066 54068 54068 (bash) 0 0 115843072 362 /bin/bash -c /usr/java/jdk1.8.0_60/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN  -Xmx6144m -Djava.io.tmpdir=/u10/yarn/nm/usercache/username/appcache/application_1463557283514_0017/container_1463557283514_0017_01_018425/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/u04/yarn/container-logs/application_1463557283514_0017/container_1463557283514_0017_01_018425 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 1x.4x.3x.9x 47528 attempt_1463557283514_0017_r_000008_1 18425 1>/u04/yarn/container-logs/application_1463557283514_0017/container_1463557283514_0017_01_018425/stdout 2>/u04/yarn/container-logs/application_1463557283514_0017/container_1463557283514_0017_01_018425/stderr
        |- 54073 54068 54068 54068 (java) 5355 6080 8573480960 1103372 /usr/java/jdk1.8.0_60/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx6144m -Djava.io.tmpdir=/u10/yarn/nm/usercache/username/appcache/application_1463557283514_0017/container_1463557283514_0017_01_018425/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/u04/yarn/container-logs/application_1463557283514_0017/container_1463557283514_0017_01_018425 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 1x.4x.3x.9x 47528 attempt_1463557283514_0017_r_000008_1 18425

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

Answer 1

我通过以下操作解决了这个问题

以前无论我做什么，如果我增加 map、reducer、资源管理器和容器内存，它使用的会超过我配置的。然后最后我改为cloudera default java。我使用的是 1.8.0_60。我已经在管理器中明确设置了这个。所以当我将这个参数以及其他更改的参数更改为 CM 默认设置时，它开始工作

Answer 2

我已经通过在 yarn-site.xml

中添加以下属性解决了这个问题

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
   <description>Whether virtual memory limits will be enforced for containers</description>
</property>
<property>
   <name>yarn.nodemanager.vmem-pmem-ratio</name>
   <value>4</value>
   <description>Ratio between virtual memory to physical memory when setting memory limits for containers</description>
</property>

添加后重启所有 hadoop 守护进程。

容器应要求终止，而运行映射作业

Container killed on request, while running mapred jobs

hadoop

mapreduce

cloudera

hadoop-yarn

hadoop2

容器应要求终止，而 运行 映射作业

Container killed on request, while running mapred jobs

hadoop

mapreduce

cloudera

hadoop-yarn

hadoop2

容器应要求终止，而运行映射作业