Not able to read parquet files in spark : java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods

Question

我正在尝试读取活泼的压缩镶木地板文件，但一直低于异常。我找不到此异常的根本原因，有人可以在这里指导我吗？

   val sparkSession: SparkSession = SparkSession.builder()
  .master("local[2]")
  .config("spark.ui.enabled",false)
  .appName("local-intellij")
  .getOrCreate()
val df = sparkSession.read.parquet("C:\data\parquet\part-00000-4ce5708f-2f50-485d-8ae4-7c5ea440fda6.c000.snappy.parquet")

我的依赖项是：

lazy val json4sVersion = "3.5.0"
lazy val json4sDeps = Seq(
"org.json4s" %% "json4s-core" % json4sVersion,
"org.json4s" %% "json4s-native" % json4sVersion,
"org.json4s" %% "json4s-ast" % json4sVersion,
"org.json4s" %% "json4s-jackson" % json4sVersion)
lazy val sparkVersionCore = "2.3.0.cloudera2"
lazy val sparkDeps = Seq(
"org.apache.spark" %% "spark-hive" % sparkVersionCore,
"org.apache.spark" %% "spark-core" % sparkVersionCore)

java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue; at org.apache.spark.sql.types.DataType$.fromJson(DataType.scala:113) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$org$apache$spark$sql$execution$datasources$parquet$ParquetFileFormat$$deserializeSchemaString.apply(ParquetFileFormat.scala:650) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$org$apache$spark$sql$execution$datasources$parquet$ParquetFileFormat$$deserializeSchemaString.apply(ParquetFileFormat.scala:650) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.org$apache$spark$sql$execution$datasources$parquet$ParquetFileFormat$$deserializeSchemaString(ParquetFileFormat.scala:650) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$readSchemaFromFooter.apply(ParquetFileFormat.scala:643) at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$readSchemaFromFooter.apply(ParquetFileFormat.scala:643)

Answer 1

清除 spark 版本 2.3.0 的 jar 版本不匹配。

据我所知，您必须使用 org.json4s json4s-jackson_2.11 3.2.11

// https://mvnrepository.com/artifact/org.json4s/json4s-jackson
libraryDependencies += "org.json4s" %% "json4s-jackson" % "3.2.11"

据我所知，这个条目不是必需的我认为一旦你在 sbt 中提到 spark 版本就会自动下载。只需尝试删除该条目并查看...如果它不起作用，请添加上述条目。

Not able to read parquet files in spark : java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods

Not able to read parquet files in spark : java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods

scala

maven

sbt

apache-spark

parquet