PySpark 中返回所有空值的列总和 SQL
Sum of column returning all null values in PySpark SQL
我是 Spark 的新手，这可能是一个简单的问题。
我有一个名为 sql_left 的 SQL，其格式为：
这是使用 sql_left.take(1) 生成的示例数据：
[Row(REPORT_ID='2016-30-15/08/2019', Stats Area='2 Metropolitan', Suburb='GREENACRES', Postcode=5086, LGA Name='CITY OF PORT ADELAIDE ENFIELD', Total Units=3, Total Cas=0, Total Fats=0, Total SI=0, Total MI=0, Year=2016, Month='November', Day='Wednesday', Time='01:20 am', Area Speed=50, Position Type='Not Divided', Horizontal Align='Straight road', Vertical Align='Level', Other Feat='Not Applicable', Road Surface='Sealed', Moisture Cond='Dry', Weather Cond='Not Raining', DayNight='Night', Crash Type='Hit Parked Vehicle', Unit Resp=1, Entity Code='Driver Rider', CSEF Severity='1: PDO', Traffic Ctrls='No Control', DUI Involved=None, Drugs Involved=None, ACCLOC_X=1331135.04, ACCLOC_Y=1677256.22, UNIQUE_LOC=13311351677256, REPORT_ID='2016-30-15/08/2019', Unit No=2, No Of Cas=0, Veh Reg State='UNKNOWN', Unit Type='Motor Vehicle - Type Unknown', Veh Year='XXXX', Direction Of Travel='East', Sex=None, Age=None, Lic State=None, Licence Class=None, Licence Type=None, Towing='Unknown', Unit Movement='Parked', Number Occupants='000', Postcode=None, Rollover=None, Fire=None)]
注：年龄栏有'XXX'、'NUll'等整数值如023,034等
printSchema 将 Age、Total Cas 显示为整数。
我尝试了以下代码来首先连接两个表：
sql_left = spark.sql('''
SELECT *
FROM sql_crash c Left JOIN sql_units u ON c.REPORT_ID=u.REPORT_ID''')
sql_left.createOrReplaceTempView("mytable")
下面的代码生成总 Cas:
sql_result = spark.sql('''select concat_ws(' ', Day, Month,Year,Time) as Date_Time,Age,"Licence Type","Unit Type",Sex,COALESCE(sum("Total Cas"),0) as Total_casualities from mytable where Suburb in ('ADELAIDE','ADELAIDE AIRPORT','NORTH ADELAIDE','PORT ADELAIDE') Group by Date_Time, Age,"Licence Type","Unit Type",Sex order by Total_casualities desc''')
sql_result.show(20,truncate=False)
我得到的输出低于总和为 0。
+--------------------------------+---+------------+---------+-------+-----------------+
|Date_Time |Age|Licence Type|Unit Type|Sex |Total_casualities|
+--------------------------------+---+------------+---------+-------+-----------------+
|Friday December 2016 02:45 pm |XXX|Licence Type|Unit Type|Unknown|0.0 |
|Saturday September 2017 06:35 pm|023|Licence Type|Unit Type|Male |0.0 |
+--------------------------------+---+------------+---------+-------+-----------------+
我尝试了多种选择，但没有任何效果。
我这里的主要问题是，如果我使用 COALESCE(sum("Total Cas"),0)，Total_casualities 会为所有行返回 0.0。如果我不使用 COALESCE，它会将值显示为 NULL。
非常感谢您的帮助。
与其在 double-quotes("Total Cas") 中指定 Total Cas，不如在反引号中提及它。
i.e. `Total Cas`
注意：中间带有space的列名需要用反引号指定。正如您在引号下提到的那样，它认为它是一个字符串，这就是您没有得到总和的原因。此外，对于其他列（如 Licence Type、Unit Type），它显示的与字符串相同，而不是它的值。希望你明白了。
sql_result = spark.sql('''select concat_ws(' ', Day, Month,Year,Time) as Date_Time,Age,`Licence Type`,`Unit Type`,Sex,**sum(`Total Cas`)** as Total_casualities from mytable where Suburb in ('ADELAIDE','ADELAIDE AIRPORT','NORTH ADELAIDE','PORT ADELAIDE') Group by Date_Time, Age,`Licence Type`,`Unit Type`,Sex order by Total_casualities desc''')
我是 Spark 的新手，这可能是一个简单的问题。
我有一个名为 sql_left 的 SQL，其格式为：
这是使用 sql_left.take(1) 生成的示例数据：
[Row(REPORT_ID='2016-30-15/08/2019', Stats Area='2 Metropolitan', Suburb='GREENACRES', Postcode=5086, LGA Name='CITY OF PORT ADELAIDE ENFIELD', Total Units=3, Total Cas=0, Total Fats=0, Total SI=0, Total MI=0, Year=2016, Month='November', Day='Wednesday', Time='01:20 am', Area Speed=50, Position Type='Not Divided', Horizontal Align='Straight road', Vertical Align='Level', Other Feat='Not Applicable', Road Surface='Sealed', Moisture Cond='Dry', Weather Cond='Not Raining', DayNight='Night', Crash Type='Hit Parked Vehicle', Unit Resp=1, Entity Code='Driver Rider', CSEF Severity='1: PDO', Traffic Ctrls='No Control', DUI Involved=None, Drugs Involved=None, ACCLOC_X=1331135.04, ACCLOC_Y=1677256.22, UNIQUE_LOC=13311351677256, REPORT_ID='2016-30-15/08/2019', Unit No=2, No Of Cas=0, Veh Reg State='UNKNOWN', Unit Type='Motor Vehicle - Type Unknown', Veh Year='XXXX', Direction Of Travel='East', Sex=None, Age=None, Lic State=None, Licence Class=None, Licence Type=None, Towing='Unknown', Unit Movement='Parked', Number Occupants='000', Postcode=None, Rollover=None, Fire=None)]
注：年龄栏有'XXX'、'NUll'等整数值如023,034等
printSchema 将 Age、Total Cas 显示为整数。
我尝试了以下代码来首先连接两个表：
sql_left = spark.sql('''
SELECT *
FROM sql_crash c Left JOIN sql_units u ON c.REPORT_ID=u.REPORT_ID''')
sql_left.createOrReplaceTempView("mytable")
下面的代码生成总 Cas:
sql_result = spark.sql('''select concat_ws(' ', Day, Month,Year,Time) as Date_Time,Age,"Licence Type","Unit Type",Sex,COALESCE(sum("Total Cas"),0) as Total_casualities from mytable where Suburb in ('ADELAIDE','ADELAIDE AIRPORT','NORTH ADELAIDE','PORT ADELAIDE') Group by Date_Time, Age,"Licence Type","Unit Type",Sex order by Total_casualities desc''')
sql_result.show(20,truncate=False)
我得到的输出低于总和为 0。
+--------------------------------+---+------------+---------+-------+-----------------+
|Date_Time |Age|Licence Type|Unit Type|Sex |Total_casualities|
+--------------------------------+---+------------+---------+-------+-----------------+
|Friday December 2016 02:45 pm |XXX|Licence Type|Unit Type|Unknown|0.0 |
|Saturday September 2017 06:35 pm|023|Licence Type|Unit Type|Male |0.0 |
+--------------------------------+---+------------+---------+-------+-----------------+
我尝试了多种选择，但没有任何效果。
我这里的主要问题是，如果我使用 COALESCE(sum("Total Cas"),0)，Total_casualities 会为所有行返回 0.0。如果我不使用 COALESCE，它会将值显示为 NULL。
非常感谢您的帮助。
与其在 double-quotes("Total Cas") 中指定 Total Cas，不如在反引号中提及它。
i.e. `Total Cas`
注意：中间带有space的列名需要用反引号指定。正如您在引号下提到的那样，它认为它是一个字符串，这就是您没有得到总和的原因。此外，对于其他列（如 Licence Type、Unit Type），它显示的与字符串相同，而不是它的值。希望你明白了。
sql_result = spark.sql('''select concat_ws(' ', Day, Month,Year,Time) as Date_Time,Age,`Licence Type`,`Unit Type`,Sex,**sum(`Total Cas`)** as Total_casualities from mytable where Suburb in ('ADELAIDE','ADELAIDE AIRPORT','NORTH ADELAIDE','PORT ADELAIDE') Group by Date_Time, Age,`Licence Type`,`Unit Type`,Sex order by Total_casualities desc''')