如何在 Spark- Sql 中解析 'MM/dd/yyyy HH:mm:ss.SSS' 格式的微秒

Question

 spark.sql("select case when trim(map('OPT OUT',1,'OPT IN',0,'',0)[coalesce(upper(program_1),'')]) == trim(num_fg) then trim(from_unixtime(unix_timestamp(upd_dt,'MM/dd/yyyy HH:mm:ss.SSS'), 'yyyy-MM-dd HH:mm:ss.sss')) else Now() end as upd_dt from input").show(false)

输入

val sample = Seq(("OPT OUT","1","07/21/2020 09:09:09.382")).toDF("program_1","num_fg", "upd_dt")

在上面的书面查询中，微秒 'sss' 没有返回我们提供的输入。

如果输入是 07/21/2020 09:09:09.382，它返回 07/21/2020 09:09:09.009，但预期结果是 07/21/2020 09:09:09.382 [无论我们在输入中给出多少微秒，它都应该显示在输出中]。

Answer 1

这会有所帮助

 import java.sql.Timestamp
 import java.time.LocalDateTime
 import java.time.format.DateTimeFormatter

 def dateWithMilliSeconds (rawDate:String) = {
  val format = DateTimeFormatter.ofPattern("MM/dd/yyyy HH:mm:ss.SSS")
  Timestamp.valueOf(LocalDateTime.parse(rawDate.trim,format))
 }

spark.udf.register("dateWithMilliSeconds",dateWithMilliSeconds _)

spark.sql("select 
             case when trim(map('OPT OUT',1,'OPT IN',0,'',0 [coalesce(upper(program_1),'')]) == trim(num_fg) 
             then dateWithMilliSeconds(upd_dt) 
             else Now() end as upd_dt 
           from input
           ").show(false)

OUTPUT:
+-----------------------+
|upd_dt                 |
+-----------------------+
|2020-07-21 09:09:09.382|
+-----------------------+

如何在 Spark- Sql 中解析 'MM/dd/yyyy HH:mm:ss.SSS' 格式的微秒

How to resolve Micro second in 'MM/dd/yyyy HH:mm:ss.SSS' format in Spark- Sql

sql

dataframe

apache-spark

apache-spark-sql