在 Spark 中区分价值和滞后

Question

我有一个 SparkR DataFrame，如下所示。我想创建一个 monthdiff 列，它是 dates 之间的月份，按每个 name 分组。我怎样才能做到这一点？

#Set up data frame
team <- data.frame(name = c("Thomas", "Thomas", "Thomas", "Thomas", "Bill", "Bill", "Bill"),
  dates = c('2017-01-05', '2017-02-23', '2017-03-16', '2017-04-08', '2017-06-08','2017-07-24','2017-09-05'))
#Create Spark DataFrame
team <- createDataFrame(team)
#Convert dates to date type
team <- withColumn(team, 'dates', cast(team$dates, 'date'))

这是我迄今为止尝试过的方法，但都导致了错误：

team <- agg(groupBy(team, 'name'), monthdiff=c(NA, months_between(team$dates, lag(team$dates))))
team <- agg(groupBy(team, 'name'), monthdiff=months_between(team$dates, lag(team$dates)))
team <- agg(groupBy(team, 'name'), monthdiff=months_between(select(team, 'dates'), lag(select(team, 'dates'))))

预期输出：

name    | dates     | monthdiff
-------------------------------
Thomas  |2017-01-05 |  NA
Thomas  |2017-02-23 |  1
Thomas  |2017-03-16 |  1
Thomas  |2017-04-08 |  1
Bill    |2017-06-08 |  NA
Bill    |2017-07-24 |  1
Bill    |2017-09-05 |  2

Answer 1

基于此，我修改了 SparkR 的代码以获得答案。

#Create 'lagdates' variable with lag of dates
window <- orderBy(windowPartitionBy("name"), team$dates)
team <- withColumn(team, 'lagdates', over(lag(team$dates), window))

#Get months_between dates and lagdates
team <- withColumn(team, 'monthdiff', round(months_between(team$dates, team$lagdates)))

name  | dates      | lagdates  | monthdiff
------------------------------------------
Bill  | 2017-06-08 |null       | null
Bill  | 2017-07-24 |2017-06-08 |    2
Bill  | 2017-09-05 |2017-07-24 |    1
Thomas| 2017-01-05 |null       | null
Thomas| 2017-02-23 |2017-01-05 |    2
Thomas| 2017-03-16 |2017-02-23 |    1
Thomas| 2017-04-08 |2017-03-16 |    1

在 Spark 中区分价值和滞后

Getting difference between value and its lag in Spark

apache-spark

pyspark

spark-dataframe

sparkr