当字符串也包含 UDF 时，如何在 spark 中将 List[String] 传递给 select 或 selectExpr

Question

我有如下列表[String]

val colList = List(verifyLength($"col1", lit(0), lit(0), lit("RJ"), lit(9)).as("col1"), verifyLength($"col2", lit(0), lit(0), lit("RJ"), lit(16)).as("col2"))

当我尝试将 select 应用于如下所示的数据框时

val selDataDf = df.select(colList.head, colList.tail: _*)

我收到异常消息 - org.apache.spark.sql.AnalysisException: cannot resolve '`verifyLength($"col1", lit(0), lit(0), lit("RJ"), lit(9)).as("col1")`' given input columns:

如果我像下面这样传递字符串，它会给我 select 语句的结果。

val selDataDf = df.select(verifyLength($"col1", lit(0), lit(0), lit("RJ"), lit(9)).as("col1"), verifyLength($"col2", lit(0), lit(0), lit("RJ"), lit(16)).as("col2"))

Note: "verifyLength" is an user defined udf.

Answer 1

我找到了它不起作用的原因，

您可以像下面这样传递 List[String]：

val detailsList = List(verifyLength(col1, 0, 0, 'RJ', 9) as col1, verifyLength(col2, 0, 0, 'RJ', 9) as col2)
spark.udf.register("verifyLength", verifyLengthCheck _)
 val df1= df.selectExpr(detailsList:_*)

请按以下url:enter link description here

当字符串也包含 UDF 时，如何在 spark 中将 List[String] 传递给 select 或 selectExpr

How to pass List[String] to select or selectExpr in spark when string contains an UDF as well

scala

apache-spark

apache-spark-sql

hadoop2