Livy Batch API - 无法获得罐子

Question

我正在尝试使用 Livy Batch API 调用我的 Spark 应用程序。

curl -X POST -d '{"file": "file:///path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory": "8g","executorCores": 2,"numExecutors": 2,"queue": "default","name": "Livy_sample_prog"}' -H "Content-Type: application/json" http://serverip:8998/batches/

POST/batches 失败并出现以下错误：

"Warning: Local jar /path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar does not exist, skipping.",
  "java.lang.ClassNotFoundException: App",

以下是我到目前为止尝试过的各种方法

1.Uploaded jar 到 HDFS

curl -X POST -d '{"file": "hdfs://path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar","className": "App","args": ["abc"],"jars": [],"driverMemory": "16g","executorMemory": "8g","executorCores": 2,"numExecutors": 2,"queue": "default","name": "Livy_sample_prog"}' -H "Content-Type: application/json" http://serverip:8998/batches/

异常：

"Warning: Skip remote jar hdfs://path to file/SampleSparkProject-0.0.2-SNAPSHOT.jar.",
  "java.lang.ClassNotFoundException: App"

2.added livy.file.local-dir-whitelist 作为包含 jar 文件的目录

3.changed file:/// 到本地:/

我已多次验证文件是否存在，每次提供的路径均有效。

我现在别无选择。请帮助找到解决方案。

谢谢！！

Answer 1

我能够将这项工作作为 yarn-cluster 而不是 yarn 交给我的主人。

这是我使用的 --data

{
    "file": "local:/path/to/file/SampleSparkProject-0.0.2-SNAPSHOT.jar",
    "proxyUser": "nrsapp",
    "className": "App",
    "args": [
        "abc"
    ],
    "name": "Livy_sample_prog",
    "conf": {
        "spark.master": "yarn-cluster",
        "spark.deployMode": "client"
    }
}

Livy Batch API - 无法获得罐子

Livy Batch API - not able to get the jar

apache-spark

livy