Spark-Solr 错误 java.lang.ClassNotFoundException：org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParameter

Question

有没有人试过spark-solrhttps://github.com/lucidworks/spark-solr

当我运行这个

val options = Map("collection" -> "DS", "zkhost" -> "url:2181,url:2181,url:2181")
val df = spark.read.format("solr").options(options).load

然后我得到这个错误

  at solr.DefaultSource.createRelation(DefaultSource.scala:14)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:317)
  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:167)
  ... 49 elided
Caused by: java.lang.ClassNotFoundException: org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParameter
  at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 54 more

我的设置是 Ubuntu 18、Solr 7.4、Spark 2.4.0、Scala 2.11.12

Answer 1

项目中缺少 solrj 库。

添加 solrj 库，因为 nit 应该适合您。

Spark-Solr 错误 java.lang.ClassNotFoundException：org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParameter

Spark-Solr error java.lang.ClassNotFoundException: org.apache.solr.client.solrj.io.stream.expr.StreamExpressionParameter

solr

scala

apache-spark