Oozie Resolving Jar 版本

Question

我有一个 oozie 作业运行 yarn scheduler。

我的 oozie 作业中有两个动作是 spark 作业，然后是 java 作业。

我的 spark assmbly jar 中有 jackson 2.6 版，java assembly jar 中有 jackson 2.9。现在我的 spark 异常失败

>     java.lang.Exception: Incompatible Jackson version: 2.9.3
>       at com.sps.analytics.batch.bi.BatchApp.executeQueries(BatchApp.scala:73)
>       at com.sps.analytics.batch.bi.BatchApp$.main(BatchApp.scala:32)

如何强制我的 spark 应用程序 jar 使用 2.6 版本

// https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5" force()

// https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-annotations
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.6.5" force()

// https://mvnrepository.com/artifact/com.fasterxml.jackson.module/jackson-module-scala
dependencyOverrides += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.5" force()

// https://mvnrepository.com/artifact/com.fasterxml.jackson.dataformat/jackson-dataformat-yaml
libraryDependencies += "com.fasterxml.jackson.dataformat" % "jackson-dataformat-yaml" % "2.6.5" force()

这就是我的火花build.sbt

Answer 1

将 Java 操作设为子工作流程有助于解决问题

Oozie Resolving Jar 版本

Oozie Resolving Jar versions

hadoop

amazon-emr

oozie

hadoop-yarn

apache-spark