PySpark - 从 UDF 获取行索引
PySpark - Get index of row from UDF
我有一个数据框,我需要获取特定行的行号/索引。我想添加一个新行,使其包含 Letter 以及行 number/index 例如。 "A - 1","B - 2"
#sample data
a= sqlContext.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
输出
+------+---------+
|Letter|distances|
+------+---------+
| A| 20|
| B| 30|
| D| 80|
+------+---------+
我希望新的输出是这样的,
+------+---------------+
|Letter|distances|index|
+------+---------------+
| A| 20|A - 1|
| B| 30|B - 2|
| D| 80|D - 3|
+------+---------------+
这是我一直在做的一个功能
def cate(letter):
return letter + " - " + #index
a.withColumn("index", cate(a["Letter"])).show()
这应该有效
df = spark.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
df.createOrReplaceTempView("df")
spark.sql("select concat(Letter,' - ',row_number() over (order by Letter)) as num, * from df").show()
+-----+------+---------+
| num|Letter|distances|
+-----+------+---------+
|A - 1| A| 20|
|B - 2| B| 30|
|D - 3| D| 80|
+-----+------+---------+
既然您想使用 UDF(仅)实现结果,我们试试这个
from pyspark.sql.functions import udf, monotonically_increasing_id
from pyspark.sql.types import StringType
#sample data
a= sqlContext.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
def cate(letter, idx):
return letter + " - " + str(idx)
cate_udf = udf(cate, StringType())
a = a.withColumn("temp_index", monotonically_increasing_id())
a = a.\
withColumn("index", cate_udf(a.Letter, a.temp_index)).\
drop("temp_index")
a.show()
输出为:
+------+---------+--------------+
|Letter|distances| index|
+------+---------+--------------+
| A| 20| A - 0|
| B| 30|B - 8589934592|
| D| 80|D - 8589934593|
+------+---------+--------------+
我有一个数据框,我需要获取特定行的行号/索引。我想添加一个新行,使其包含 Letter 以及行 number/index 例如。 "A - 1","B - 2"
#sample data
a= sqlContext.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
输出
+------+---------+
|Letter|distances|
+------+---------+
| A| 20|
| B| 30|
| D| 80|
+------+---------+
我希望新的输出是这样的,
+------+---------------+
|Letter|distances|index|
+------+---------------+
| A| 20|A - 1|
| B| 30|B - 2|
| D| 80|D - 3|
+------+---------------+
这是我一直在做的一个功能
def cate(letter):
return letter + " - " + #index
a.withColumn("index", cate(a["Letter"])).show()
这应该有效
df = spark.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
df.createOrReplaceTempView("df")
spark.sql("select concat(Letter,' - ',row_number() over (order by Letter)) as num, * from df").show()
+-----+------+---------+
| num|Letter|distances|
+-----+------+---------+
|A - 1| A| 20|
|B - 2| B| 30|
|D - 3| D| 80|
+-----+------+---------+
既然您想使用 UDF(仅)实现结果,我们试试这个
from pyspark.sql.functions import udf, monotonically_increasing_id
from pyspark.sql.types import StringType
#sample data
a= sqlContext.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
def cate(letter, idx):
return letter + " - " + str(idx)
cate_udf = udf(cate, StringType())
a = a.withColumn("temp_index", monotonically_increasing_id())
a = a.\
withColumn("index", cate_udf(a.Letter, a.temp_index)).\
drop("temp_index")
a.show()
输出为:
+------+---------+--------------+
|Letter|distances| index|
+------+---------+--------------+
| A| 20| A - 0|
| B| 30|B - 8589934592|
| D| 80|D - 8589934593|
+------+---------+--------------+