如何增加 yarn application 的并行度

Question

我正在尝试在 EMR Spark 上运行多个 yarn 应用程序，但我无法一次运行超过 5 个应用程序。

我正在为 Spark 集群使用以下配置：

大师 = r5.2xlarge

工人 = r5.12xlarge 384 GB 内存 48 个虚拟内核部署模式=集群

JSON

{
        "Classification":"spark-defaults",
        "ConfigurationProperties":{
          "spark.executor.extraJavaOptions": "-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'",
          "spark.driver.extraJavaOptions": "-XX:+UseG1GC -XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -XX:InitiatingHeapOccupancyPercent=35 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p'",
          "spark.scheduler.mode":"FIFO",
          "spark.eventLog.enabled":"true",
          "spark.serializer":"org.apache.spark.serializer.KryoSerializer",
          "spark.dynamicAllocation.enabled":"false",
          "spark.executor.heartbeatInterval":"60s",
          "spark.network.timeout": "800s",
          "spark.executor.cores": "5",
          "spark.driver.cores": "5",
          "spark.executor.memory": "37000M",
          "spark.driver.memory": "37000M",
          "spark.yarn.executor.memoryOverhead": "5000M",
          "spark.yarn.driver.memoryOverhead": "5000M",
          "spark.executor.instances": "17",
          "spark.default.parallelism": "170",
          "spark.yarn.scheduler.reporterThread.maxFailures": "5",
          "spark.storage.level": "MEMORY_AND_DISK_SER",
          "spark.rdd.compress": "true",
          "spark.shuffle.compress": "true",
          "spark.shuffle.spill.compress": "true"
        }
      }

如何增加 EMR Spark 中并行运行ning Yarn 应用程序的数量？

Answer 1

看看集群主节点上的Yarn ui 运行ning。集群中是否使用了所有 CPU 和所有内存？增加并发性通常意味着运行ning 中的每个单独应用程序只能使用集群的一小部分。此外，因为您已禁用动态执行程序分配并将执行程序的数量设置为 17，您可能一次只能运行一个 spark 应用程序。

如何增加 yarn application 的并行度

How to increase yarn application parallelism

amazon-emr

hadoop-yarn

pyspark