sparklyr下的值匹配

Question

我尝试使用 sparklyr 进行值匹配：

spark_parquet %>% filter(customer_id %in% spark_unique_customer_ids)

但是，我收到以下错误：

Error in UseMethod("escape") : no applicable method for 'escape' applied to an object of class "c('tbl_spark', 'tbl_sql', 'tbl_lazy', 'tbl')"

有什么解决这个问题的建议吗？

Answer 1

最好用semi_join:

spark_parquet <- copy_to(
  sc, 
  tibble(customer_id = c(1, 2, 3), value = c(-1, 0, 1))
)
spark_unique_customer_ids <- copy_to(sc, tibble(customer_id = c(1, 3)))

spark_parquet %>% semi_join(spark_unique_customer_ids, by = "customer_id")

# Source: spark<?> [?? x 2]
  customer_id value
*       <dbl> <dbl>
1           1    -1
2           3     1

尽管 SQL API 也应该有效：

spark_parquet %>%  sdf_register("spark_parquet")
spark_unique_customer_ids %>% sdf_register("spark_unique_customer_ids")

sc %>% spark_session() %>% 
  invoke(
    "sql", 
    "SELECT * FROM spark_parquet 
     WHERE customer_id IN (
       SELECT customer_id FROM spark_unique_customer_ids)") %>% 
   sdf_register()

# Source: spark<?> [?? x 2]
  customer_id value
*       <dbl> <dbl>
1           1    -1
2           3     1

sparklyr下的值匹配

Value matching under sparklyr

r

matching

dplyr

apache-spark

sparklyr