如何使用 org.apache.spark.launcher.SparkLauncher 设置队列

Question

如果我使用 spark-submit 命令行将 spark 任务提交给 yarn，我将能够使用 --queue myqueuename

设置队列

完整命令为

/myserver/spark-2.2.0-bin-hadoop2.4/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--executor-memory 4G \
--executor-cores 2 \
--queue myqueuename \
--class task.MyTask \
 "/my/jar/path/spark-app-full.jar"  \
--input  /data/input/path \
--output  /data/output/path/

但是，java 如何使用 SparkLauncher 设置队列。我想以编程方式启动 spark 任务。

到目前为止我的 java 代码

SparkAppHandle handle = new SparkLauncher()
        .setSparkHome("/myserver/spark-2.2.0-bin-hadoop2.4")
        .setAppResource(jarPath)
        .setMainClass("task.MyTask")
        .setMaster("yarn")
        .setDeployMode("cluster")
        .setConf(SparkLauncher.EXECUTOR_MEMORY, "8G")
        .setConf(SparkLauncher.EXECUTOR_CORES, "4")
        .addAppArgs("--input",
                inputPath,
                "--output",
                outputPath)
        .startApplication(taskListener);

我想知道如何使用 SparkLauncher

在 java 代码中指定队列

Answer 1

根据 https://spark.apache.org/docs/latest/running-on-yarn.html，setConf("spark.yarn.queue", "myqueuename") 的作用是：

SparkAppHandle handle = new SparkLauncher()
        .setSparkHome("/myserver/spark-2.2.0-bin-hadoop2.4")
        .setAppResource(jarPath)
        .setMainClass("task.MyTask")
        .setMaster("yarn")
        .setDeployMode("cluster")
        .setConf(SparkLauncher.EXECUTOR_MEMORY, "8G")
        .setConf(SparkLauncher.EXECUTOR_CORES, "4")
        .setConf("spark.yarn.queue", "myqueuename") // <-- SETTING A QUEUE NAME
        .addAppArgs("--input",
                inputPath,
                "--output",
                outputPath)
        .startApplication(taskListener);

如何使用 org.apache.spark.launcher.SparkLauncher 设置队列

How to set queue with org.apache.spark.launcher.SparkLauncher

java

apache-spark

spark-launcher