Spark：如何从对象文件中读取 CompactBuffer？

Question

我正在从目标文件中读取以下结构：

(String, CompactBuffer(person1, person2, person3 ...) )

如果我尝试这样阅读：

val input = sc.objectFile[(String, ListBuffer[Person])]("inputFile.txt")

val myData = input.map { t =>
  val myList = t._2
  for (p <- myList) {
    println(p.toString())
  }
  t
}

我收到以下错误：

java.lang.ClassCastException: org.apache.spark.util.collection.CompactBuffer cannot be cast to scala.collection.mutable.ListBuffer

但是，我也不能在读取对象文件时使用 CompactBuffer:

val input = sc.objectFile[(String, CompactBuffer[Person])]("inputFile.txt")

Eclipse 只会告诉我：

class CompactBuffer in package collection cannot be accessed in package 
 org.apache.spark.util.collection

那么如何从对象文件中读取这样的 CompactBuffer 呢？谢谢！

Answer 1

查看https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala我们可以看到CompactBuffer是Seq的子class，所以试试val input = sc.objectFile[(String, Seq[Person])]("inputFile.txt")

Spark：如何从对象文件中读取 CompactBuffer？

Spark: how to read CompactBuffer from an objectFile?

scala

object-files

sequencefile

apache-spark