无法在 AWS 上的 sparkR 中创建根目录

Question

在将 sparkR 连接到 AWS 集群的第一步中，我遇到了一个问题：

我无法在 Rstudio 中创建 sparkcontext ('sc') -

> .libPaths( c( .libPaths(), '/root/spark/R/lib') ) 

> Sys.setenv(SPARK_HOME = '/root/spark')

> Sys.setenv(PATH = paste(Sys.getenv(c("PATH")), '/root/spark/bin', sep=':'))

> library(SparkR)

> sc <- sparkR.init('ec2-54-159-90-154.compute-1.amazonaws.com:7077')

我得到：

    15/07/16 13:20:55 INFO SparkContext: Running Spark version 1.4.0
    15/07/16 13:20:55 INFO SecurityManager: Changing view acls to: rootroot1
    15/07/16 13:20:55 INFO SecurityManager: Changing modify acls to: rootroot1
    15/07/16 13:20:55 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rootroot1); users with modify permissions: Set(rootroot1)
    15/07/16 13:20:56 INFO Slf4jLogger: Slf4jLogger started
    15/07/16 13:20:56 INFO Remoting: Starting remoting
    15/07/16 13:20:56 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.101.183.7:36597]
    15/07/16 13:20:56 INFO Utils: Successfully started service 'sparkDriver' on port 36597.
    15/07/16 13:20:56 INFO SparkEnv: Registering MapOutputTracker
    15/07/16 13:20:56 INFO SparkEnv: Registering BlockManagerMaster
    15/07/16 13:20:56 ERROR Utils: Failed to create local root dir in /mnt/spark. Ignoring this directory.
    15/07/16 13:20:56 ERROR Utils: Failed to create local root dir in /mnt2/spark. Ignoring this directory.
    15/07/16 13:20:56 ERROR DiskBlockManager: Failed to create any local dir.
    15/07/16 13:20:56 INFO Utils: Shutdown hook called
    Error in readTypedObject(con, type) : 
      Unsupported type for deserialization

我试过很多用户

如有任何帮助，我们将不胜感激发件人 Z

Answer 1

您需要将 /mnt/spark 和 /mnt2/spark 的权限更改为所有用户都可写（默认情况下只能由 root 用户写入）。您可以通过运行 chmod a+w /mnt/spark 来完成此操作。要让 RStudio 在 EC2 上工作，您还需要执行几个步骤。 http://blog.godatadriven.com/spark-rstudio.html 是一个很好的教程

无法在 AWS 上的 sparkR 中创建根目录

cannot create root directory in sparkR on AWS

amazon-web-services

rstudio

sparkr