根据一列获取不同的行

Question

+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|ID_NOTIFICATION|ID_ENTITE|ID_ENTITE_GARANTE|CD_ETAT|DT_ETAT            |CD_ANOMALIE|CD_TYPE_DESTINATAIRE|CD_TYPE_EVENEMENT   |CD_SYS_APPELANT|TYP_MVT|DT_DEBUT           |DT_FIN             |
+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|3110305        |GNE      |GNE              |AT     |2019-06-12 00:03:14|null       |null                |REL_CP_ULTIME_PAPIER|SIGMA          |C      |2019-06-12 00:03:22|2019-06-12 00:03:32|
|3110305        |GNE      |GNE              |AN     |2019-06-12 00:03:28|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:08:43|
|3110305        |GNE      |GNE              |AN     |2019-06-12 00:03:28|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:10:06|
|3110305        |GNE      |GNE              |AN     |2019-06-12 15:10:02|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:10:51|
|3110305        |GNE      |GNE              |AN     |2019-06-12 15:10:02|017        |IDGRC               |REL_CP_ULTIME_PAPIER|SIGMA          |M      |2019-06-12 00:03:22|2019-06-12 15:11:35|

有没有办法获取每个不同 CD_ETAT 列的一行？在这种情况下，它将是前两行。

类似于 this SQL solution 但请在 Scala 中使用 DF 函数。谢谢

Answer 1

您可以 window 函数与 partitionBy CD_ETAT 并选择 orderBy 以获得第一个

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._

val window = Window.partitionBy("CD_ETAT").orderBy("DT_ETAT")

df.withColumn("row_num", row_number().over(window))
  .filter($"row_num" === 1)
  .drop("row_num")

输出：

+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|ID_NOTIFICATION|ID_ENTITE|ID_ENTITE_GARANTE|CD_ETAT|            DT_ETAT|CD_ANOMALIE|CD_TYPE_DESTINATAIRE|   CD_TYPE_EVENEMENT|CD_SYS_APPELANT|TYP_MVT|           DT_DEBUT|             DT_FIN|
+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+
|        3110305|      GNE|              GNE|     AT|2019-06-12 00:03:14|       null|                null|REL_CP_ULTIME_PAPIER|          SIGMA|      C|2019-06-12 00:03:22|2019-06-12 00:03:32|
|        3110305|      GNE|              GNE|     AN|2019-06-12 00:03:28|        017|               IDGRC|REL_CP_ULTIME_PAPIER|          SIGMA|      M|2019-06-12 00:03:22|2019-06-12 15:08:43|
+---------------+---------+-----------------+-------+-------------------+-----------+--------------------+--------------------+---------------+-------+-------------------+-------------------+

根据一列获取不同的行

Get distinct rows based on one column

scala

distinct

dataframe

apache-spark