使用 R 将 databricks 中的 spark dataframe 写入 azure data lake store

Question

我想使用 R save/write/upload 从 databricks 到 azure data lack store 文件夹的 spark dataframe。我找到了以下 python 代码。

spark_df.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").mode("overwrite").save('...path to azure data lake store folder')

你能给我推荐一个与此代码等效的 SparkR 吗？

Answer 1

这应该是：

spark_df %>% 
  coalesce(1L) %>%          # Same as coalesce(1).
  write.df(                 # Generic writer, because there is no csv specific one
    "...path to azure...",  # Path as before 
     source = "csv",        # Since 2.0 you don't need com.databricks 
     mode = "overwrite", 
     header = "true"        # All ... are used as options
  )

使用 R 将 databricks 中的 spark dataframe 写入 azure data lake store

Write spark dataframe from databricks onto the azure data lack store using R

r

apache-spark

sparkr