Hive Warehouse Connector + Spark = 签名者信息与同一包中其他类的签名者信息不匹配

Question

我正在尝试在 hdp 3.1 上使用 hive warehouse connector 和 spark，即使使用最简单的示例（如下）也出现异常。导致问题的 class：JaninoRuntimeException - 在 org.codehaus.janino:janino:jar:3.0.8（spark_sql 的依赖项）和 com.hortonworks.hive:hive-warehouse-connector_2.11:jar 中。

我试图从 spark_sql 中排除 janino 库，但这导致从 janino 中丢失其他 classes。我需要 hwc 来实现新功能。

有人遇到同样的错误吗？任何想法如何处理它？

我遇到错误：

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66)
Caused by: java.lang.SecurityException: class "org.codehaus.janino.JaninoRuntimeException"'s signer information does not match signer information of other classes in the same package
    at java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
    at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access0(URLClassLoader.java:73)
    at java.net.URLClassLoader.run(URLClassLoader.java:368)
    at java.net.URLClassLoader.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access0(URLClassLoader.java:73)
    at java.net.URLClassLoader.run(URLClassLoader.java:368)
    at java.net.URLClassLoader.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:197)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:36)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:1321)
    at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3277)
    at org.apache.spark.sql.Dataset$$anonfun$head.apply(Dataset.scala:2489)
    at org.apache.spark.sql.Dataset$$anonfun$head.apply(Dataset.scala:2489)
    at org.apache.spark.sql.Dataset$$anonfun.apply(Dataset.scala:3259)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
    at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
    at org.apache.spark.sql.Dataset.head(Dataset.scala:2489)
    at org.apache.spark.sql.Dataset.take(Dataset.scala:2703)
    at org.apache.spark.sql.Dataset.showString(Dataset.scala:254)
    at org.apache.spark.sql.Dataset.show(Dataset.scala:723)
    at org.apache.spark.sql.Dataset.show(Dataset.scala:682)
    at org.apache.spark.sql.Dataset.show(Dataset.scala:691)
    at Main$.main(Main.scala:15)
    at Main.main(Main.scala)
    ... 5 more

我的 sbt 文件：

name := "testHwc"

version := "0.1"

scalaVersion := "2.11.11"

resolvers += "Hortonworks repo" at "http://repo.hortonworks.com/content/repositories/releases/"

libraryDependencies += "org.apache.hadoop" % "hadoop-aws" % "3.1.1.3.1.0.0-78"

// https://mvnrepository.com/artifact/com.hortonworks.hive/hive-warehouse-connector
libraryDependencies += "com.hortonworks.hive" %% "hive-warehouse-connector" % "1.0.0.3.1.0.0-78"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.2.3.1.0.0-78"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.2.3.1.0.0-78"

以及源代码：

import com.hortonworks.hwc.HiveWarehouseSession
import org.apache.spark.sql.SparkSession

object Main {
  def main(args: Array[String]): Unit = {

    val ss = SparkSession.builder()
      .config("spark.sql.hive.hiveserver2.jdbc.url", "nnn")
      .master("local[*]").getOrCreate()

    import ss.sqlContext.implicits._

    val rdd = ss.sparkContext.makeRDD(Seq(1, 2, 3, 4, 5, 6, 7))

    rdd.toDF("col1").show()
    val hive = HiveWarehouseSession.session(ss).build()
  }
}

Answer 1

经过一些调查，我发现错误的存在取决于 classpath 中库的顺序。

由于未知原因，当我在 IntelliJ IDEA 中运行这个项目时，类路径总是随机排序，应用程序随机失败和成功。

最后 - HiveWarehouseConnector jar in classpath 应该是 after Spark Sql jar.

更新

正如 this 回答中所建议的那样 - IntelliJ IDEA 中的顺序可以在依赖项选项卡中更改。

否则 - 我无法为 IntelliJ 解决这个问题 - 顺序总是随机的，但是当我在 IntelliJ 之外执行程序时 - 我设置了我需要的顺序。

Answer 2

在maven项目中，我是这样解决的

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.11</artifactId>
    <version>2.3.1</version>
    <exclusions>
        <exclusion>
           <groupId>org.codehaus.janino</groupId>
           <artifactId>janino</artifactId>
        </exclusion>
    </exclusions>
</dependency>



<dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>janino</artifactId>
    <version>3.0.10</version>
</dependency>

Hive Warehouse Connector + Spark = 签名者信息与同一包中其他类的签名者信息不匹配

Hive Warehouse Connector + Spark = signer information does not match signer information of other classes in the same package

hive

hortonworks-data-platform

apache-spark

hdp

Hive Warehouse Connector + Spark = 签名者信息与同一包中其他 类 的签名者信息不匹配

Hive Warehouse Connector + Spark = signer information does not match signer information of other classes in the same package

hive

hortonworks-data-platform

apache-spark

hdp

Hive Warehouse Connector + Spark = 签名者信息与同一包中其他类的签名者信息不匹配