如何查看我的 mapreduce 作业是否在多个节点上执行？

Question

我在集群中有运行个 Map Reduce 作业。我在 HDInsight 上用 2 个名称节点和 4 个数据节点创建了一个作业。

我没有设置map任务和reduce任务的数量。执行我的 mapreduce 作业后，我得到如下结果。

我注意到启动的 reduce 任务数是 1.Dose 这意味着我的作业只在一个节点上执行？如何查看此作业中使用了多少个节点？

File System Counters
                FILE: Number of bytes read=2209390166
                FILE: Number of bytes written=3314494070
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                WASB: Number of bytes read=1084887535
                WASB: Number of bytes written=1205106549
                WASB: Number of read operations=0
                WASB: Number of large read operations=0
                WASB: Number of write operations=0
        Job Counters
                Launched map tasks=2
                Launched reduce tasks=1
                Rack-local map tasks=2
                Total time spent by all maps in occupied slots (ms)=148221
                Total time spent by all reduces in occupied slots (ms)=302038
                Total time spent by all map tasks (ms)=148221
                Total time spent by all reduce tasks (ms)=151019
                Total vcore-seconds taken by all map tasks=148221
                Total vcore-seconds taken by all reduce tasks=151019
                Total megabyte-seconds taken by all map tasks=113833728
                Total megabyte-seconds taken by all reduce tasks=231965184
        Map-Reduce Framework
                Map input records=3820642
                Map output records=3820642
                Map output bytes=1092815223
                Map output materialized bytes=1104695065
                Input split bytes=286
                Combine input records=0
                Combine output records=0
                Reduce input groups=200
                Reduce shuffle bytes=1104695065
                Reduce input records=3820642
                Reduce output records=3820642
                Spilled Records=11461926
                Shuffled Maps =2
                Failed Shuffles=0
                Merged Map outputs=2
                GC time elapsed (ms)=16229
                CPU time spent (ms)=225140
                Physical memory (bytes) snapshot=1377296384
                Virtual memory (bytes) snapshot=5068787712
                Total committed heap usage (bytes)=1175453696
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1084887106
        File Output Format Counters
                Bytes Written=68771556

Answer 1

是的，你将reduce数设置为1，输出reduce为1，你可以看到它Map input records=3820642 地图输出记录=3820642 地图输出字节数=1092815223 映射输出物化字节=1104695065 输入拆分字节=286 合并输入记录=0 合并输出记录=0 减少输入组=200 减少随机播放字节数=1104695065 减少输入记录=3820642 减少输出记录=3820642

如何查看我的 mapreduce 作业是否在多个节点上执行？

How to see if my mapreduce job is executed on multiple nodes?

hadoop

mapreduce

azure-hdinsight