在 Spark 中通过 DIMSUM 进行余弦相似度

Question

我有一个非常简单的代码来尝试余弦相似度：

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.linalg.distributed.{MatrixEntry,   CoordinateMatrix, RowMatrix}

val rows= Array(((1,2,3,4,5),(1,2,3,4,5),(1,2,4,5,8),(3,4,1,2,7),(7,7,7,7,7)))
val mat = new RowMatrix(rows)

val simsPerfect = mat.columnSimilarities()
val simsEstimate = mat.columnSimilarities(0.8)

我运行 Amazon AWS 上的此代码具有 Spark 1.5，但是我在最后两行收到以下消息： "Erroe: value columnSimilarities is not a memeber of org.apache.spark.rdd.RDD[(int,int)]"

你能帮忙解决这个问题吗？

Answer 1

我找到了答案。我需要将矩阵转换为 RDD。这是正确的代码：

import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.linalg.distributed.{MatrixEntry, CoordinateMatrix, RowMatrix}
import org.apache.spark.rdd._
import org.apache.spark.mllib.linalg._


def matrixToRDD(m: Matrix): RDD[Vector] = {
val columns = m.toArray.grouped(m.numRows)
val rows = columns.toSeq.transpose // Skip this if you want a column-major RDD.
val vectors = rows.map(row => new DenseVector(row.toArray))
sc.parallelize(vectors)
}

val dm: Matrix = Matrices.dense(5, 5,Array(1,2,3,4,5,1,2,3,4,5,1,2,4,5,8,3,4,1,2,7,7,7,7,7,7))
val rows = matrixToRDD(dm)
val mat = new RowMatrix(rows)
val simsPerfect = mat.columnSimilarities()
val simsEstimate = mat.columnSimilarities(0.8)

println("Pairwise similarities are: " + simsPerfect.entries.collect.mkString(", "))

println("Estimated pairwise similarities are: " +     simsEstimate.entries.collect.mkString(", "))

干杯

在 Spark 中通过 DIMSUM 进行余弦相似度

Cosine Similarity via DIMSUM in Spark

scala

apache-spark

similarity

trigonometry