SBT Assembly 插件出错

Question

我写了如下sbt文件

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0",
  "org.apache.avro" % "avro" % "1.7.7",
  "org.apache.avro" % "avro-mapred" % "1.7.7"
)

mainClass := Some("com.test.Foo")

我在项目文件夹中还有以下 assembly.sbt 文件

resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")

当我这样做时 sbt assembly 我得到一个巨大的错误列表

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:

Answer 1

我能够解决问题。实际上不需要构建一个 fat jar，因为 "spark-submit" 工具无论如何都会在类路径中包含所有内容。

因此构建 jar 文件的正确方法是

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1" % "provided",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0" % "provided",
  "org.apache.avro" % "avro" % "1.7.7" % "provided",
  "org.apache.avro" % "avro-mapred" % "1.7.7" % "provided"
)

mainClass := Some("com.test.Foo")

Answer 2

1。使用 MergeStrategy，参见 sbt-assembly

2。排除重复的jar，如：

    lazy val hbaseLibSeq = Seq(
    ("org.apache.hbase" % "hbase" % hbaseVersion).
      excludeAll(
        ExclusionRule(organization = "org.slf4j"),
        ExclusionRule(organization = "org.mortbay.jetty"),
        ExclusionRule(organization = "javax.servlet")),
    ("net.java.dev.jets3t" % "jets3t" % "0.6.1" % "provided").
      excludeAll(ExclusionRule(organization = "javax.servlet"))
  )

3。使用提供的范围

显示依赖关系树：

➜ cat ~/.sbt/0.13/plugins/plugins.sbt

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.5") addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

➜ cat ~/.sbt/0.13/global.sbt

net.virtualvoid.sbt.graph.Plugin.graphSettings

➜ sbt 依赖图

SBT Assembly 插件出错

SBT Assembly plugin Errors out

scala

sbt

sbt-assembly

1。使用 MergeStrategy，参见 sbt-assembly

2。排除重复的jar，如：

3。使用提供的范围

显示依赖关系树：