YARN MapReduce 内存不足
YARN MapReduce has not enough RAM
我正在尝试在 Yarn MapReduce 中启动我的应用程序。我在 4 个主机（1 个资源管理器和 4 个节点管理器）中设置了 MapReduce。每个主机有 2 个内核和 4GB RAM。
当我 运行 我的应用程序时，它由于缺少 RAM [1] 而中止。我应该如何设置 Yarn MapReduce 以使作业不会 运行 超出 RAM？
[1] 我遇到的错误。
2016-02-09 16:01:41,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.6 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:44,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:47,616 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:50,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:53,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 291.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:56,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.1 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:59,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.3 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
(...)
2016-02-09 16:02:01,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/syslog
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stdout
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stderr
2016-02-09 16:02:01,250 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001
2016-02-09 16:02:02,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455032058147_0001_01_000001
2016-02-09 16:03:10,146 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:11,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:12,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:13,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
您需要配置分配给 YARN 的内存量及其在 yarn-site.xml 中的容器：
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
检查此示例 yarn-site.xml file. As you're running MapReduce jobs you've also to set similar properties in the mapred-site.xmlfile. This article may help you - link。
我正在尝试在 Yarn MapReduce 中启动我的应用程序。我在 4 个主机（1 个资源管理器和 4 个节点管理器）中设置了 MapReduce。每个主机有 2 个内核和 4GB RAM。
当我 运行 我的应用程序时，它由于缺少 RAM [1] 而中止。我应该如何设置 Yarn MapReduce 以使作业不会 运行 超出 RAM？
[1] 我遇到的错误。
2016-02-09 16:01:41,607 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.6 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:44,612 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:47,616 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:50,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 289.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:53,626 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 291.7 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:56,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.1 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
2016-02-09 16:01:59,643 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 1639 for container-id container_1455032058147
_0001_01_000001: 295.3 MB of 2 GB physical memory used; 2.7 GB of 4.2 GB virtual memory used
(...)
2016-02-09 16:02:01,173 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/syslog
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stdout
2016-02-09 16:02:01,174 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001/container_1455032058147_0001_01_000001/stderr
2016-02-09 16:02:01,250 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /home/ubuntu/Programs/hadoop-yarn-medusa/logs/userlogs/application_1455032058147_0001
2016-02-09 16:02:02,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1455032058147_0001_01_000001
2016-02-09 16:03:10,146 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:11,147 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:12,148 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 16:03:13,149 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: ip-172-30-0-125/172.30.0.125:8025. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
您需要配置分配给 YARN 的内存量及其在 yarn-site.xml 中的容器：
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
检查此示例 yarn-site.xml file. As you're running MapReduce jobs you've also to set similar properties in the mapred-site.xmlfile. This article may help you - link。