如何设置所有字段为空的空结构,spark中为空
How to set an empty struct with all fields null, null in spark
我有这个数据框:
+----+--------------------------------+
|name|dates |
+----+--------------------------------+
|A |[[1994, 12, 11], [,,]] |
|B |[[1994, 12, 11], [1994, 12, 15]]|
+----+--------------------------------+
使用此架构:
root
|-- name: string (nullable = true)
|-- dates: struct (nullable = true)
| |-- start_date: struct (nullable = true)
| | |-- year: integer (nullable = true)
| | |-- month: integer (nullable = true)
| | |-- day: integer (nullable = true)
| |-- end_date: struct (nullable = true)
| | |-- year: integer (nullable = true)
| | |-- month: integer (nullable = true)
| | |-- day: integer (nullable = true)
我想将其作为输出
当 end_date
中的所有字段都为 null 时,将结束日期设置为 null
+----+--------------------------------+
|name|dates |
+----+--------------------------------+
|A |[[1994, 12, 11],] |
|B |[[1994, 12, 11], [1994, 12, 15]]|
+----+--------------------------------+
您可以通过从现有属性重新创建一个新结构来更新结构列 dates
,并使用 when
表达式检查所有 end_dates
属性是否为空:
val df2 = df.withColumn(
"dates",
struct(
col("dates.start_date"), // keep start_date
when(
Seq("year", "month", "day")
.map(x => col(s"dates.end_date.$x").isNull)
.reduce(_ and _),
lit(null).cast("struct<year:int,month:int,day:int>")
).otherwise(col("dates.end_date")).alias("end_date") // set end_date to null if all attr are null
)
)
df2.show(false)
//+----+--------------------------------+
//|name|dates |
//+----+--------------------------------+
//|A |[[1994, 12, 11],] |
//|B |[[1994, 12, 11], [1994, 12, 25]]|
//+----+--------------------------------+
我有这个数据框:
+----+--------------------------------+
|name|dates |
+----+--------------------------------+
|A |[[1994, 12, 11], [,,]] |
|B |[[1994, 12, 11], [1994, 12, 15]]|
+----+--------------------------------+
使用此架构:
root
|-- name: string (nullable = true)
|-- dates: struct (nullable = true)
| |-- start_date: struct (nullable = true)
| | |-- year: integer (nullable = true)
| | |-- month: integer (nullable = true)
| | |-- day: integer (nullable = true)
| |-- end_date: struct (nullable = true)
| | |-- year: integer (nullable = true)
| | |-- month: integer (nullable = true)
| | |-- day: integer (nullable = true)
我想将其作为输出
当 end_date
中的所有字段都为 null 时,将结束日期设置为 null
+----+--------------------------------+
|name|dates |
+----+--------------------------------+
|A |[[1994, 12, 11],] |
|B |[[1994, 12, 11], [1994, 12, 15]]|
+----+--------------------------------+
您可以通过从现有属性重新创建一个新结构来更新结构列 dates
,并使用 when
表达式检查所有 end_dates
属性是否为空:
val df2 = df.withColumn(
"dates",
struct(
col("dates.start_date"), // keep start_date
when(
Seq("year", "month", "day")
.map(x => col(s"dates.end_date.$x").isNull)
.reduce(_ and _),
lit(null).cast("struct<year:int,month:int,day:int>")
).otherwise(col("dates.end_date")).alias("end_date") // set end_date to null if all attr are null
)
)
df2.show(false)
//+----+--------------------------------+
//|name|dates |
//+----+--------------------------------+
//|A |[[1994, 12, 11],] |
//|B |[[1994, 12, 11], [1994, 12, 25]]|
//+----+--------------------------------+