spark2-提交多个包的抛出错误（--packages）

Question

我正在尝试在 CDH 5.16 集群上提交以下 Spark2 作业，它只采用 --packages 选项的第一个参数并为第二个参数抛出错误

spark2-submit --packages com.databricks:spark-xml_2.11:0.4.1, com.databricks:spark-csv_2.11:1.5.0 /path/to/python-script


Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR com.databricks:spark-csv_2.11:1.5.0 with URI com.databricks. Please specify a class through --class.
        at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657)
        at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:224)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116)
        at org.apache.spark.deploy.SparkSubmit$$anon$$anon.<init>(SparkSubmit.scala:911)
        at org.apache.spark.deploy.SparkSubmit$$anon.parseArguments(SparkSubmit.scala:911)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
        at org.apache.spark.deploy.SparkSubmit$$anon.doSubmit(SparkSubmit.scala:924)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

运行这个作业在 CDH5.16 集群中并且 spark 安装了 Spark2 CSD

提前致谢。

Answer 1

包裹之间不要给space

spark2-submit --packages com.databricks:spark-xml_2.11:0.4.1,com.databricks:spark-csv_2.11:1.5.0 /path/to/python-script

spark2-提交多个包的抛出错误（--packages）

spark2-submit throwing error with multiple packages (--packages)

apache-spark

cloudera-cdh

spark-submit