过滤 pyspark 数据框
Filtering a pyspark dataframe
我正在尝试 select 根据某些规则从 pyspark 数据帧中获取一些值。在 pyspark 中出现异常。
from pyspark.sql import functions as F
df.select(df.card_key,F.when((df.tran_sponsor = 'GAMES') & (df.location_code = '9145'),'ENTERTAINMENT').when((df.tran_sponsor = 'XYZ') & (df.location_code = '123'),'eBOOKS').when((df.tran_sponsor = 'XYZ') & (df.l_code.isin(['123', '234', '345', '456', '567', '678', '789', '7878', '67', '456']) ),'FINANCE').otherwise(df.tran_sponsor)).show()
我遇到以下异常。你能给点建议吗？
File "", line 1
df.select(df.card_key,F.when((df.tran_sponsor = 'GAMES') & (df.location_code = '9145'),'ENTERTAINMENT').when((df.tran_sponsor = 'XYZ') & (df.location_code = '123'),'eBOOKS').when((df.tran_sponsor = 'XYZ') & (df.l_code.isin(['6001', '6002', '6003', '6004', '6005', '6006', '6007', '6008', '6009', '6010', '6011', '6012', '6013', '6014'])),'FINANCE').otherwise(df.tran_sponsor)).show()
^
SyntaxError: invalid syntax
好吧，我刚刚发现，isin 没有问题，问题出在赋值运算符上:(
df.select(df.card_key,F.when((df.tran_sponsor == 'GAMES') & (df.location_code == '9145'),'ENTERTAINMENT').when((df.tran_sponsor == 'XYZ') & (df.location_code == '123'),'eBOOKS').when((df.tran_sponsor == 'XYZ') & (df.l_code.isin(['123', '234', '345', '456', '567', '678', '789', '7878', '67', '456']) ),'FINANCE').otherwise(df.tran_sponsor)).show()
效果很好，感谢大家的努力
我正在尝试 select 根据某些规则从 pyspark 数据帧中获取一些值。在 pyspark 中出现异常。
from pyspark.sql import functions as F
df.select(df.card_key,F.when((df.tran_sponsor = 'GAMES') & (df.location_code = '9145'),'ENTERTAINMENT').when((df.tran_sponsor = 'XYZ') & (df.location_code = '123'),'eBOOKS').when((df.tran_sponsor = 'XYZ') & (df.l_code.isin(['123', '234', '345', '456', '567', '678', '789', '7878', '67', '456']) ),'FINANCE').otherwise(df.tran_sponsor)).show()
我遇到以下异常。你能给点建议吗？
File "", line 1 df.select(df.card_key,F.when((df.tran_sponsor = 'GAMES') & (df.location_code = '9145'),'ENTERTAINMENT').when((df.tran_sponsor = 'XYZ') & (df.location_code = '123'),'eBOOKS').when((df.tran_sponsor = 'XYZ') & (df.l_code.isin(['6001', '6002', '6003', '6004', '6005', '6006', '6007', '6008', '6009', '6010', '6011', '6012', '6013', '6014'])),'FINANCE').otherwise(df.tran_sponsor)).show() ^ SyntaxError: invalid syntax
好吧，我刚刚发现，isin 没有问题，问题出在赋值运算符上:(
df.select(df.card_key,F.when((df.tran_sponsor == 'GAMES') & (df.location_code == '9145'),'ENTERTAINMENT').when((df.tran_sponsor == 'XYZ') & (df.location_code == '123'),'eBOOKS').when((df.tran_sponsor == 'XYZ') & (df.l_code.isin(['123', '234', '345', '456', '567', '678', '789', '7878', '67', '456']) ),'FINANCE').otherwise(df.tran_sponsor)).show()
效果很好，感谢大家的努力