select 来自由字符串数组 pyspark 或 python 高阶函数多个值组成的列

Question

我有一个这样的 table，我想根据列中列出的内容创建一个新列例子

df.withcolumn('good',.when('java' or 'php' isin ['booksIntereste']).lit(1).otherwise(0) )

包含 java 或 php 时的期望输出得到 1 否则 0

Answer 1

可以直接用一篇Higher Order Function-array_contains for this , additionally you can browse through this文章了解更多

数据准备

d = {
    'name':['James','Washington','Robert','Micheal'],
    'booksInterested':[['Java','C#','Python'],[],['PHP','Java'],['Java']]
}

sparkDF = sql.createDataFrame(pd.DataFrame(d))

sparkDF.show()

+----------+------------------+
|      name|   booksInterested|
+----------+------------------+
|     James|[Java, C#, Python]|
|Washington|                []|
|    Robert|       [PHP, Java]|
|   Micheal|            [Java]|
+----------+------------------+

数组包含

sparkDF = sparkDF.withColumn('good',F.array_contains(F.col('booksInterested'), 'Java'))

+----------+------------------+-----+
|      name|   booksInterested| good|
+----------+------------------+-----+
|     James|[Java, C#, Python]| true|
|Washington|                []|false|
|    Robert|       [PHP, Java]| true|
|   Micheal|            [Java]| true|
+----------+------------------+-----+

ForAll 数组包含 - 多个

sparkDF = sparkDF.withColumn('good_multiple',F.forall(F.col('booksInterested'), lambda x: x.isin(['Java','Python','PHP'])))

sparkDF.show()

+----------+------------------+-----+-------------+
|      name|   booksInterested| good|good_multiple|
+----------+------------------+-----+-------------+
|     James|[Java, C#, Python]| true|        false|
|Washington|                []|false|         true|
|    Robert|       [PHP, Java]| true|         true|
|   Micheal|            [Java]| true|         true|
+----------+------------------+-----+-------------+

select 来自由字符串数组 pyspark 或 python 高阶函数多个值组成的列

select from a column made of string array pyspark or python high order function multiple values

python

arraylist

dataframe

higher-order-functions

pyspark

数据准备

数组包含

ForAll 数组包含 - 多个