Scala 错误 Spark Streaming Kafka：对重载定义的模糊引用

Question

我正在尝试通过在我的 Spark 流模块中从外部提供偏移来创建 kafka 直接流，但是它会导致以下编译错误。

下面是创建Kafka直接流的代码

val kafkaParams = Map("metadata.broker.list" -> "kafka.brokers")
// testing only
val fromOffsets: Map[TopicPartition, Long] = Map[TopicPartition, Long]()

val kafkaStream = KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder, Array[Byte]]
    (ssc, kafkaParams, fromOffsets, (mmd: MessageAndMetadata[Array[Byte], Array[Byte]]) => mmd.message())

这是我运行遇到的编译错误。有 ideas/pointers 吗？

    ambiguous reference to overloaded definition,
both method createDirectStream in object KafkaUtils of type (jssc: org.apache.spark.streaming.api.java.JavaStreamingContext, keyClass: Class[Array[Byte]], valueClass: Class[Array[Byte]], keyDecoderClass: Class[kafka.serializer.DefaultDecoder], valueDecoderClass: Class[kafka.serializer.DefaultDecoder], recordClass: Class[Array[Byte]], kafkaParams: java.util.Map[String,String], fromOffsets: java.util.Map[kafka.common.TopicAndPartition,Long], messageHandler: org.apache.spark.api.java.function.Function[kafka.message.MessageAndMetadata[Array[Byte],Array[Byte]],Array[Byte]])org.apache.spark.streaming.api.java.JavaInputDStream[Array[Byte]]
and  method createDirectStream in object KafkaUtils of type (ssc: org.apache.spark.streaming.StreamingContext, kafkaParams: Map[String,String], fromOffsets: Map[kafka.common.TopicAndPartition,Long], messageHandler: kafka.message.MessageAndMetadata[Array[Byte],Array[Byte]] => Array[Byte])(implicit evidence: scala.reflect.ClassTag[Array[Byte]], implicit evidence: scala.reflect.ClassTag[Array[Byte]], implicit evidence: scala.reflect.ClassTag[kafka.serializer.DefaultDecoder], implicit evidence: scala.reflect.ClassTag[kafka.serializer.DefaultDecoder], implicit evidence: scala.reflect.ClassTag[Array[Byte]])org.apache.spark.streaming.dstream.InputDStream[Array[Byte]]
match expected type ?
[ERROR]     val kafkaStream = KafkaUtils.createDirectStream[Array[Byte], Array[Byte], DefaultDecoder, DefaultDecoder, Array[Byte]]

Answer 1

请使用kafka.common.TopicAndPartition代替org.apache.kafka.common.TopicPartition。

Scala 错误 Spark Streaming Kafka：对重载定义的模糊引用

Scala error Spark Streaming Kafka : ambiguous reference to overloaded definition

scala

apache-spark

spark-streaming