IgniteRdd读取parquet文件的savepairs方法

Question

为了读取镶木地板文件并将项目存储到缓存中，我做了一个小实现。所以我写道：

val df= sqlContext.read.
        parquet(hdfsFolder).
        select("a","b", "c", "d", "e", "f")
val columnsSeq= Seq("a","b", "c", "d", "e", "f")

val values = df.map(row => (row.getAs[String]("a"), row.getValuesMap(columnsSeq))).
  groupByKey(1024).
  map(row => (row._1 , row._2.toList.asJava ))

//put them into cache

val igniteContext = new IgniteContext(sc, cacheConfigPath)

val sharedRdd = igniteContext.fromCache(cacheName)

sharedRdd.savePairs(values)

但是最后一行"sharedRdd.savePairs(values)"给出了编译错误：

found : org.apache.spark.rdd.RDD[(String, java.util.List[Map[String,Nothing]])] required: org.apache.spark.rdd.RDD[(Nothing, Nothing)] Note: (String, java.util.List[Map[String,Nothing]]) >: (Nothing, Nothing), but class RDD is invariant in type T. You may wish to define T as -T instead. (SLS 4.5) sharedRdd.savePairs(values)

我找不到任何方法来克服这个错误。

有什么想法吗？

Answer 1

您应该通过正确输入来创建 IgniteRDD：

val sharedRdd = igniteContext.fromCache[String, java.util.List[Map[String,Nothing]]](cacheName)

IgniteRdd读取parquet文件的savepairs方法

IgniteRdd's savepairs method for reading parquet files

caching

apache-spark

rdd

ignite