使 2 个数据帧中的结构数组相同（Java Spark）

Question

我有两个数据框 (Dataset<Row>)，它们具有相同的列，但结构数组的顺序不同。

df1:

root
|-- root: string (nullable = false)
|-- array_nested: array (nullable = false)
|    |-- element: struct (containsNull = true)
|    |    |-- array_id: integer (nullable = false)
|    |    |-- array_value: string (nullable = false)

+----+------------+
|root|array_nested|
+----+------------+
|One |[[1, 1-One]]|
+----+------------+

df2:

root
|-- root: string (nullable = false)
|-- array_nested: array (nullable = false)
|    |-- element: struct (containsNull = true)
|    |    |-- array_value: string (nullable = false)
|    |    |-- array_id: integer (nullable = false)


+----+------------+
|root|array_nested|
+----+------------+
|Two |[[2-Two, 2]]|
+----+------------+

我想让模式相同，但是当我尝试我的方法时，它会生成数组并在以后生成：

List<Column> updatedStructNames = new ArrayList<>();
updatedStructNames.add(col("array_nested.array_id"));
updatedStructNames.add(col("array_nested.array_value"));

Column[] updatedStructNameArray = updatedStructNames.toArray(new Column[0]);
Dataset<Row> df3 = df2.withColumn("array_nested", array(struct(updatedStructNameArray)));

它将生成这样的架构：

root
 |-- root: string (nullable = false)
 |-- array_nested: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- array_id: array (nullable = false)
 |    |    |    |-- element: integer (containsNull = true)
 |    |    |-- array_value: array (nullable = false)
 |    |    |    |-- element: string (containsNull = true)

+----+----------------+
|root|array_nested    |
+----+----------------+
|Two |[[[2], [2-Two]]]|
+----+----------------+

如何实现相同的架构？

Answer 1

您可以使用 transform 函数来更新 array_nested 列的结构元素：

Dataset < Row > df3 = df2.withColumn(
    "array_nested",
    expr("transform(array_nested, x -> struct(x.array_id as array_id, x.array_value as array_value))")
);