无法使用命令 spark-submit 启动 jar Spark

Question

我尝试使用 tSystem 组件 TALEND 启动 jar Spark，但出现错误：

命令 tSystem :

"cmd /c spark-submit --class reprisesComptesExistants.PhasePreliminaire.main --master local[5] C:/Users/A661758/dataintegrationRepriseExistant/target/scala-2.11/dataintegrationRepriseExistant-assembly-0.1.0-SNAPSHOT.jar"

错误：

Warning: Skip remote jar C:/Users/A661758/dataintegrationRepriseExistant/target/scala-2.11/dataintegrationRepriseExistant-assembly-0.1.0-SNAPSHOT.jar.
java.lang.ClassNotFoundException: reprisesComptesExistants.PhasePreliminaire
    at java.net.URLClassLoader.findClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.ClassLoader.loadClass(Unknown Source)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Unknown Source)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:695)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
[statistics] disconnected
Job test1 terminé à 13:54 16/06/2017. [Code sortie=0]

Answer 1

请确保映射 class 名称正确。
应该是 package.classname 就像 com.company.test.Classname.
"com.company.test" 是你的包，Classname 是你的 class.

如果您使用的是 talend bigdata platform V6.2.1，我建议您使用带有 spark 的 bigdata batch job 而不是 standarad job。

Answer 2

local: - a URI starting with local:/ is expected to exist as a local file on each worker node. This means that no network IO will be incurred, and works well for large files/JARs that are pushed to each worker, or shared via NFS, GlusterFS, etc.

尝试像这样在文件 uri 之前添加 local:/ :

local:/C:/Users/A661758/dataintegrationRepriseExistant/target/scala-2.11/dataintegrationRepriseExistant-assembly-0.1.0-SNAPSHOT.jar

或者要摆脱 windows fs 路径问题，只需将您的 jar 放入 HDFS 并使用此 synthax :

hdfs://hostname:9000/path/dataintegrationRepriseExistant-assembly-0.1.0-SNAPSHOT.jar

无法使用命令 spark-submit 启动 jar Spark

Can not launch jar Spark with command spark-submit

talend

apache-spark