如何在转换为 RDD 的情况下在 spark 数据集中保存嵌套或 JSON 对象?
How to save nested or JSON object in spark Dataset with converting to RDD?
我正在处理 spark 代码,我必须将多个列值保存为对象格式并将结果保存到 mongodb
给定数据集
|---|-----|------|----------|
|A |A_SRC|Past_A|Past_A_SRC|
|---|-----|------|----------|
|a1 | s1 | a2 | s2 |
我尝试过的
val ds1 = Seq(("1", "2", "3","4")).toDF("a", "src", "p_a","p_src")
val recordCol = functions.to_json(Seq($"a", $"src", $"p_a",$"p_src"),struct("a", "src", "p_a","p_src")) as "A"
ds1.select(recordCol).show(truncate = false)
给我这样的结果
+-----------------------------------------+
|A |
+-----------------------------------------+
|{"a":"1","src":"2","p_a":"3","p_src":"4"}|
+-----------------------------------------+
我期待
+-----------------------------------------+
|A |
+-----------------------------------------+
|{"source":"1","value":"2","p_source":"4","p_value":"3"}|
+-----------------------------------------+
如何更改对象类型中除列名以外的键。在 java ?
中使用地图
您可以在 struct
列中传递 as
,这样它将被保存为您传递的名称。
Dataset<Row> tstDS = spark.read().format("csv").option("header", "true").load("/home/exa9/Documents/SparkLogs/y.csv");
tstDS.show();
/****
+---+-----+------+----------+
| A|A_SRC|Past_A|Past_A_SRC|
+---+-----+------+----------+
| a1| s1| a2| s2|
+---+-----+------+----------+
****/
tstDS.withColumn("A",
functions.to_json(
functions.struct(
functions.col("A").as("source"),
functions.col("A_SRC").as("value"),
functions.col("Past_A").as("p_source"),
functions.col("Past_A_SRC").as("p_value")
))
)
.select("A")
.show(false);
/****
+-----------------------------------------------------------+
|A |
+-----------------------------------------------------------+
|{"source":"a1","value":"s1","p_source":"a2","p_value":"s2"}|
+-----------------------------------------------------------+
****/
我正在处理 spark 代码,我必须将多个列值保存为对象格式并将结果保存到 mongodb
给定数据集
|---|-----|------|----------|
|A |A_SRC|Past_A|Past_A_SRC|
|---|-----|------|----------|
|a1 | s1 | a2 | s2 |
我尝试过的
val ds1 = Seq(("1", "2", "3","4")).toDF("a", "src", "p_a","p_src")
val recordCol = functions.to_json(Seq($"a", $"src", $"p_a",$"p_src"),struct("a", "src", "p_a","p_src")) as "A"
ds1.select(recordCol).show(truncate = false)
给我这样的结果
+-----------------------------------------+
|A |
+-----------------------------------------+
|{"a":"1","src":"2","p_a":"3","p_src":"4"}|
+-----------------------------------------+
我期待
+-----------------------------------------+
|A |
+-----------------------------------------+
|{"source":"1","value":"2","p_source":"4","p_value":"3"}|
+-----------------------------------------+
如何更改对象类型中除列名以外的键。在 java ?
中使用地图您可以在 struct
列中传递 as
,这样它将被保存为您传递的名称。
Dataset<Row> tstDS = spark.read().format("csv").option("header", "true").load("/home/exa9/Documents/SparkLogs/y.csv");
tstDS.show();
/****
+---+-----+------+----------+
| A|A_SRC|Past_A|Past_A_SRC|
+---+-----+------+----------+
| a1| s1| a2| s2|
+---+-----+------+----------+
****/
tstDS.withColumn("A",
functions.to_json(
functions.struct(
functions.col("A").as("source"),
functions.col("A_SRC").as("value"),
functions.col("Past_A").as("p_source"),
functions.col("Past_A_SRC").as("p_value")
))
)
.select("A")
.show(false);
/****
+-----------------------------------------------------------+
|A |
+-----------------------------------------------------------+
|{"source":"a1","value":"s1","p_source":"a2","p_value":"s2"}|
+-----------------------------------------------------------+
****/