计算数据框中列的平均字符串长度

Question

我不知道如何使用 Scala 计算数据框中任何列的平均字符串长度。我已经能够轻松地为数字列执行以下操作

val avgDF = df.dtypes.filter(x => x._2 == "DoubleType").map(ct =>avg(col(ct._1))).toList

Answer 1

import org.apache.spark.sql.functions._

val avgDF = df.agg(mean(length(col("yourColumn"))))

Answer 2

val findLength = udf { (ColValue: String) => ColValue.size }

  myData.dtypes.filter(x=>x._2=="StringType").foreach(f=>
  myData.select(avg(findLength(col(f._1)))).show()      
  )

示例数据

Name|Age|email
Hari|12|hary@h0otmail.ocm
Hari|12|hary@h0otmail.ocm
Hari|12|hary@h0otmail.ocm
Hari|12|hary@h0otmail.ocm
Hasasasi|12|hary@h0otmail.in

输出

+-------------------+
|AVG(scalaUDF(Name))|
+-------------------+
|                4.8|
+-------------------+


+--------------------+
|AVG(scalaUDF(email))|
+--------------------+
|                16.8|
+--------------------+

计算数据框中列的平均字符串长度

Calculating average string length of a Column in Dataframes

scala

dataframe

apache-spark

spark-dataframe