Spark 2.x scala 2.1.1 自定义编码器 class 类型不匹配

Question

我正在使用此代码在 scala 2.1.1 中为 spark 2.x 定义编码器：

import org.apache.spark.sql.Encoder
import org.apache.spark.sql.types.StructType

class LogProcessorMessageEncoder extends Encoder[LogProcessorMessage] {
    override def schema: StructType = null

    override def clsTag: Class[LogProcessorMessage] = classOf[LogProcessorMessage]
}

object LogProcessorMessage {}

class LogProcessorMessage extends Serializable {}

IntelliJ 说没问题，但是重写 clsTag 方法会导致编译器错误：

Error:(20, 15) overriding method clsTag in trait Encoder of type => scala.reflect.ClassTag[dvn.aast.telemetry.scala.stream.LogProcessorMessage];
method clsTag has incompatible type
override def clsTag: Class[LogProcessorMessage] = classOf[LogProcessorMessage]

Answer 1

clsTag 必须是 ClassTag 而不是 Class:

class LogProcessorMessageEncoder extends Encoder[LogProcessorMessage] {
  override def clsTag: ClassTag[LogProcessorMessage] = ClassTag(classOf[LogProcessorMessage])
  override def schema: StructType = null 
}