来自 2 个数据帧的 Spark scala 列级别不匹配

Question

我有 2 个数据框

val df1 = Seq((1, "1","6"), (2, "10","8"), (3, "6","4")).toDF("id", "value1","value2")
val df2 = Seq((1, "1","6"), (2, "5","4"), (4, "3","1")).toDF("id", "value1","value2")

我想找出列级别的差异输出应该看起来像

id,value1_df1,value1_df2,diff_value1,value2_df1,value_df2,diff_value2
1, 1        ,1           ,  0         , 6         ,6         ,0
2, 10       ,5           ,  5         , 8         ,4         ,4
3, 6        ,3           ,  1         , 4         ,1         ,3

同样，我有 100 列，并且想计算 2 个数据帧中同一列之间的差异列是动态的

Answer 1

也许这会有所帮助：

  val spark = SparkSession.builder.appName("Test").master("local[*]").getOrCreate();

  import spark.implicits._

  var df1 = Seq((1, "1", "6"), (2, "10", "8"), (3, "6", "4")).toDF("id", "value1", "value2")
  var df2 = Seq((1, "1", "6"), (2, "5", "4"), (3, "3", "1")).toDF("id", "value1", "value2")

  df1.columns.foreach(column => {
    df1 = df1.withColumn(column, df1.col(column).cast(IntegerType))
  })
  df2.columns.foreach(column => {
    df2 = df2.withColumn(column, df2.col(column).cast(IntegerType))
  })

  df1 = df1.withColumnRenamed("id", "df1_id")
  df2 = df2.withColumnRenamed("id", "df2_id")

  df1.show()
  df2.show()

所以到目前为止你有两个数据帧 value_x,value_y,value_z 并且继续......

df1:

+------+------+------+
|df1_id|value1|value2|
+------+------+------+
|     1|     1|     6|
|     2|    10|     8|
|     3|     6|     4|
+------+------+------+

df2:

+------+------+------+
|df2_id|value1|value2|
+------+------+------+
|     1|     1|     6|
|     2|     5|     4|
|     3|     3|     1|
+------+------+------+

现在我们要根据id加入他们：

  var df3 = df1.alias("df1").join(df2.alias("df2"), $"df1.df1_id" === $"df2.df2_id")

最后，我们将获取 df1/df2 上的所有列（*重要的是它们将具有相同的列） - 没有 id，并创建一个新的差异列：

  df1.columns.tail.foreach(col => {
    val new_col_name = s"${col}-diff"
    val df_a_col = s"df1.${col}"
    val df_b_col = s"df2.${col}"
    df3 = df3.withColumn(new_col_name, df3.col(df_a_col) - df3.col(df_b_col))
  })

  df3.show()

结果：

+------+------+------+------+------+------+-----------+-----------+
|df1_id|value1|value2|df2_id|value1|value2|value1-diff|value2-diff|
+------+------+------+------+------+------+-----------+-----------+
|     1|     1|     6|     1|     1|     6|          0|          0|
|     2|    10|     8|     2|     5|     4|          5|          4|
|     3|     6|     4|     3|     3|     1|          3|          3|
+------+------+------+------+------+------+-----------+-----------+

这是结果，它是动态的，因此您可以根据需要添加 valueX。

来自 2 个数据帧的 Spark scala 列级别不匹配

Spark scala column level mismatches from 2 dataframes

scala

difference

apache-spark

apache-spark-sql