spark-atlas-connector: "SparkCatalogEventProcessor-thread" class 未发现异常

Question

按照 spark-atlas-connector 的说明进行操作后。运行在 spark

中创建 table 的简单代码时出现以下错误

Spark2 2.3.1 图集 1.0.0

批处理命令是：

spark-submit --jars /home/user/spark-atlas-connector/spark-atlas-connector-assembly/target/spark-atlas-connector-assembly-0.1.0-SNAPSHOT.jar
--conf spark.extraListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker
--conf spark.sql.queryExecutionListeners=com.hortonworks.spark.atlas.SparkAtlasEventTracker
--conf spark.sql.streaming.streamingQueryListeners=com.hortonworks.spark.atlas.SparkAtlasStreamingQueryEventTracker
--files /home/user/atlas-application.properties
--master local
/home/user/SparkAtlas/test.py

Exception in thread "SparkCatalogEventProcessor-thread" java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/catalog/ExternalCatalogWithListener at com.hortonworks.spark.atlas.sql.SparkCatalogEventProcessor.process(SparkCatalogEventProcessor.scala:36) at com.hortonworks.spark.atlas.sql.SparkCatalogEventProcessor.process(SparkCatalogEventProcessor.scala:28) at com.hortonworks.spark.atlas.AbstractEventProcessor$$anonfun$eventProcess.apply(AbstractEventProcessor.scala:72) at com.hortonworks.spark.atlas.AbstractEventProcessor$$anonfun$eventProcess.apply(AbstractEventProcessor.scala:71) at scala.Option.foreach(Option.scala:257) at com.hortonworks.spark.atlas.AbstractEventProcessor.eventProcess(AbstractEventProcessor.scala:71) at com.hortonworks.spark.atlas.AbstractEventProcessor$$anon.run(AbstractEventProcessor.scala:38) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

提前致谢。

Answer 1

这清楚表明 jar 版本不匹配

最新的图集版本2.0.0...以下是依赖项

  <spark.version>2.4.0</spark.version>
    <atlas.version>2.0.0</atlas.version>
    <scala.version>2.11.12</scala.version>

对于 Atlas 1.0.0，请参阅 pom.xml for it... these are dependencies

 <spark.version>2.3.0</spark.version>
    <atlas.version>1.0.0</atlas.version>
    <scala.version>2.11.8</scala.version>

通过查看 link 中提到的 pom.xml 尝试使用正确版本的 jar。

注：
1) 如果您通过看到错误并下载它来添加一个 jar...和另一个地方，您将遇到障碍。建议您使用正确的版本。
2) Spark 在 Java 8+、Python 2.7+/3.4+ 和 R 3.1+ 上运行。对于 Scala API，Spark 2.3.1 使用 Scala 2.11。您将需要使用兼容的 Scala 版本 (2.11.x)。检查问题中未提及的 Scala 版本。

spark-atlas-connector: "SparkCatalogEventProcessor-thread" class 未发现异常

spark-atlas-connector: "SparkCatalogEventProcessor-thread" class not found exception

apache-spark

hortonworks-data-platform

connector

apache-atlas