使用输入向量 SparkR 按多列分组

Question

我正在使用 SparkR 2.1.0 进行数据操作

我想以编程方式按多列分组。我知道如果我单独列出它们，或者从向量中引用它们的位置，我可以按多列分组......但我希望能够将列列表作为向量传递（这样，函数会自动调整到我传递的参数数量）

虚拟数据：

 cpny <- c("Fakeco1", "Fakeco2", "Fakeco3", "Fakeco4", "Fakeco5", "Fakeco6")
 state <- c("CA", "NY", "WA", "CA", "CA", "NY")
 public <- c("Y", "Y", "N", "N", "N", "N")
 color <- c("White", "Red", "Green", "Green", "Green", "Red")
 revs <- c(400, 200, 900, 500, 200, 120)
 df <- data.frame(cpny, state, public, color, revs)
 # Convert to SparkR dataframe
 df_s <- as.DataFrame(df)

作品：

  df_grouped <- df_s %>%
  groupBy('state', 'public') %>%
  summarize(sum_Revs = sum(df_s$revs))

也有效：

  group_vars <- c('state', 'public')

  df_grouped <- df_s %>%
  groupBy(group_vars[[1]], group_vars[[2]]) %>%
  summarize(sum_Revs = sum(df_s$revs))

无效：

  group_vars <- c('state', 'public')

  df_grouped <- df_s %>%
  groupBy(group_vars) %>%
  summarize(sum_Revs = sum(df_s$revs))

任何解决方案或替代想法？

Answer 1

您可以使用 do.call() https://stat.ethz.ch/R-manual/R-devel/library/base/html/do.call.html 并将列和数据框放入列表中。以下对我有用：

cpny <- c("Fakeco1", "Fakeco2", "Fakeco3", "Fakeco4", "Fakeco5", "Fakeco6")
state <- c("CA", "NY", "WA", "CA", "CA", "NY")
public <- c("Y", "Y", "N", "N", "N", "N")
color <- c("White", "Red", "Green", "Green", "Green", "Red")
revs <- c(400, 200, 900, 500, 200, 120)
df <- data.frame(cpny, state, public, color, revs)
# Convert to SparkR dataframe
df_s <- as.DataFrame(df)  

group_vars <- c('state', 'public')


function_params <- list(df_s)
for (i in range(1:length(group_vars))) {
    function_params[[i+1]] <- group_vars[i]
}

summarized<- do.call(SparkR::groupBy, function_params) %>%  SparkR::summarize(sum_Revs = sum(df_s$revs))
SparkR::head(summarized)

使用输入向量 SparkR 按多列分组

Groupby Multiple Columns using an input vector SparkR

r

sparkr

虚拟数据：

作品：

也有效：

无效：