Spark GraphX - 如何传递和排列以过滤图形边缘？

Question

我在 Spark 2.1.0 GraphX 上使用 Scala。我有一个数组，如下所示：

scala> TEMP1Vertex.take(5)
res46: Array[org.apache.spark.graphx.VertexId] = Array(-1895512637, -1745667420, -1448961741, -1352361520, -1286348803)

如果我必须过滤边缘 table 以获得单个值，假设源 ID -1895512637

val TEMP1Edge = graph.edges.filter { case Edge(src, dst, prop) => src == -1895512637}

scala> TEMP1Edge.take(5)
res52: Array[org.apache.spark.graphx.Edge[Int]] = Array(Edge(-1895512637,-2105158920,89), Edge(-1895512637,-2020727043,3), Edge(-1895512637,-1963423298,449), Edge(-1895512637,-1855207100,214), Edge(-1895512637,-1852287689,339))

scala> TEMP1Edge.count
17/04/03 10:20:31 WARN Executor: 1 block locks were not released by TID = 1436:[rdd_36_2]
res53: Long = 126

但是当我传递一个包含一组唯一源 ID 的数组时，代码运行成功但它没有 return 任何值，如下所示：

scala> val TEMP1Edge = graph.edges.filter { case Edge(src, dst, prop) => src == TEMP1Vertex}
TEMP1Edge: org.apache.spark.rdd.RDD[org.apache.spark.graphx.Edge[Int]] = MapPartitionsRDD[929] at filter at <console>:56

scala> TEMP1Edge.take(5)
17/04/03 10:29:07 WARN Executor: 1 block locks were not released by TID = 1471:
[rdd_36_5]
res60: Array[org.apache.spark.graphx.Edge[Int]] = Array()

scala> TEMP1Edge.count
17/04/03 10:29:10 WARN Executor: 1 block locks were not released by TID = 1477:
[rdd_36_5]
res61: Long = 0

Answer 1

我假设 TEMP1Vertex 是 Array[VertexId] 类型，所以我认为您的代码应该是这样的：

val TEMP1Edge = graph.edges.filter { 
  case Edge(src, _, _) => TEMP1Vertex.contains(src) 
}

Spark GraphX - 如何传递和排列以过滤图形边缘？

Spark GraphX - How to pass and array to to filter graph edges?

arrays

scala

apache-spark

spark-graphx