NameNode heap usage 和ResourceManager heap usage 的区别（试图找到NameNode heap usage 原因）？

Question

NameNode 堆使用量和 ResourceManager 堆使用量有什么区别？我正在尝试查找 NameNode 堆使用率过高的原因。

在 ambari 仪表板中，我看到...

当运行一些 sqoop 作业时。不确定是什么导致这里的 NN 使用率如此之高（没有太多的 hadoop 管理经验）？这是异常数量吗（最近才注意到）？

此外，在 mapreduce 任务 100% 完成后，sqoop 作业似乎被冻结了异常多的时间，例如。看到...

[2020-01-31 14:00:55,193]  INFO mapreduce.JobSubmitter: number of splits:12
[2020-01-31 14:00:55,402]  INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1579648183118_1085
[2020-01-31 14:00:55,402]  INFO mapreduce.JobSubmitter: Executing with tokens: []
[2020-01-31 14:00:55,687]  INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml
[2020-01-31 14:00:55,784]  INFO impl.YarnClientImpl: Submitted application application_1579648183118_1085
[2020-01-31 14:00:55,837]  mapreduce.Job: The url to track the job: http://hw001.ucera.local:8088/proxy/application_1579648183118_1085/
[2020-01-31 14:00:55,837]  mapreduce.Job: Running job: job_1579648183118_1085
[2020-01-31 14:01:02,964]  mapreduce.Job: Job job_1579648183118_1085 running in uber mode : false
[2020-01-31 14:01:02,965]  mapreduce.Job:  map 0% reduce 0%
[2020-01-31 14:01:18,178]  mapreduce.Job:  map 8% reduce 0%
[2020-01-31 14:02:21,552]  mapreduce.Job:  map 17% reduce 0%
[2020-01-31 14:04:55,239]  mapreduce.Job:  map 25% reduce 0%
[2020-01-31 14:05:36,417]  mapreduce.Job:  map 33% reduce 0%
[2020-01-31 14:05:37,424]  mapreduce.Job:  map 42% reduce 0%
[2020-01-31 14:05:40,440]  mapreduce.Job:  map 50% reduce 0%
[2020-01-31 14:05:41,444]  mapreduce.Job:  map 58% reduce 0%
[2020-01-31 14:05:44,455]  mapreduce.Job:  map 67% reduce 0%
[2020-01-31 14:05:52,484]  mapreduce.Job:  map 75% reduce 0%
[2020-01-31 14:05:56,499]  mapreduce.Job:  map 83% reduce 0%
[2020-01-31 14:05:59,528]  mapreduce.Job:  map 92% reduce 0%
[2020-01-31 14:06:00,534] INFO mapreduce.Job:  map 100% reduce 0%

<...after some time longer than usual...>

[2020-01-31 14:10:05,446] INFO mapreduce.Job: Job job_1579648183118_1085 completed successfully

我的hadoop版本

[airflow@airflowetl root]$ hadoop version
Hadoop 3.1.1.3.1.0.0-78
Source code repository git@github.com:hortonworks/hadoop.git -r e4f82af51faec922b4804d0232a637422ec29e64
Compiled by jenkins on 2018-12-06T12:26Z
Compiled with protoc 2.5.0
From source with checksum eab9fa2a6aa38c6362c66d8df75774
This command was run using /usr/hdp/3.1.0.0-78/hadoop/hadoop-common-3.1.1.3.1.0.0-78.jar

有更多 hadoop 经验的人知道这里会发生什么吗？有什么调试建议吗？

Answer 1

Namenode堆主要由存储在HDFS中的文件块的数量决定。特别是，许多小文件或同时写入许多文件会导致大堆。

ResourceManager 与namenode 没有关联。它的堆将取决于正在积极跟踪的 YARN 作业的数量

在我维护的集群中，namenode heap 是 32G，我认为 ResourceManager 只有 8GB

NameNode heap usage 和ResourceManager heap usage 的区别（试图找到NameNode heap usage 原因）？

Difference between NameNode heap usage and ResourceManager heap usage (trying to find NameNode heap usage cause)?

hadoop

resourcemanager

namenode