import org.apache.spark.streaming.kafka._ 无法解析符号 kafka

Question

我创建了一个 spark 应用程序来与 kafka 集成并从 kafka 获取数据流。

但是，当我尝试 import import org.apache.spark.streaming.kafka._ 时发生错误 Cannot resolve symbol kafka 所以我应该怎么办做导入这个库

Answer 1

根据您的 Spark 和 Scala 版本，您需要将 spark-kafka 集成库包含到您的依赖项中。

Spark 结构化流

如果您计划使用 Spark Structured Streaming，您需要按照 here 所述将以下内容添加到您的依赖项中：

For Scala/Java applications using SBT/Maven project definitions, link your application with the following artifact:

groupId = org.apache.spark
artifactId = spark-sql-kafka-0-10_2.12
version = 3.0.1

Please note that to use the headers functionality, your Kafka client version should be version 0.11.0.0 or up. For Python applications, you need to add this above library and its dependencies when deploying your application. See the Deploying subsection below. For experimenting on spark-shell, you need to add this above library and its dependencies too when invoking spark-shell. Also, see the Deploying subsection below.

Spark 流

如果您打算使用 Spark Streaming（直接 API），您可以遵循给出的指导 here:

For Scala/Java applications using SBT/Maven project definitions, link your streaming application with the following artifact (see Linking section in the main programming guide for further information).

groupId = org.apache.spark
artifactId = spark-streaming-kafka-0-10_2.12
version = 3.0.1

import org.apache.spark.streaming.kafka._ 无法解析符号 kafka

import org.apache.spark.streaming.kafka._ Cannot resolve symbol kafka

apache-kafka

apache-spark

spark-streaming-kafka

Spark 结构化流

Spark 流