VertexRDD 给我类型不匹配错误

Question

我运行下面的代码试图在 Apache Spark 的 GraphX 中创建图形。

import org.apache.spark.SparkConf

import org.apache.spark.SparkContext

import org.apache.spark.graphx.GraphLoader 

import org.apache.spark.graphx.Graph

import org.apache.spark.rdd.RDD
import org.apache.spark.graphx.VertexId

//loads file from the array

val lines = sc.textFile("hdfs://moonshot-ha-nameservice/data/google-plus/2309.graph");

//maps lines and takes the first 21 characters of each line which is the node.

val result = lines.map( line => line.substring(0,20))

//creates a new variable with each node followed by a long .

val result2 = result.map(word => (word,1L).toLong)

//where i am getting an error

val vertexRDD: RDD[(Long,Long)] = sc.parallelize(result2)

我收到以下错误：

 error: type mismatch;

 found   : org.apache.spark.rdd.RDD[(Long, Long)]

 required: Seq[?]

Error occurred in an application involving default arguments.
         val vertexRDD: RDD[(Long, Long)] = sc.parallelize(result2)

Answer 1

首先，您的地图可以简化为以下代码：

val vertexRDD: RDD[(Long, Long)] = 
  lines.map(line => (line.substring(0, 17).toLong, 1L))

现在，对于你的错误：你不能用 RDD 调用 sc.parallelize。您的 vertexRDD 已由 result2 定义。然后，您可以使用 result2 和 EdgesRDD 创建图表：

val g = Graph(result2, edgesRDD)

或者，如果采纳我的建议：

val g = Graph(vertexRDD, edgesRDD)

VertexRDD 给我类型不匹配错误

VertexRDD giving me type mismatch error

scala

apache-spark

spark-graphx