AWS EMR - 使用 Java SDK 安装 HUE

Question

我正在尝试设置一个 StepConfig 以在我的集群上安装运行 HUE。我正在按以下方式创建步骤：

private StepConfig newInstallHueStep() {
    return new StepConfig()
            .withName("Install Hue")
            .withActionOnFailure(getDefaultActionOnFailure())
            .withHadoopJarStep(stepFactory.newScriptRunnerStep("s3://us-west-1.elasticmapreduce/libs/hue/install-hue"));
}

和

private StepConfig newRunHueStep() {
    return new StepConfig()
            .withName("Run Hue")
            .withActionOnFailure(getDefaultActionOnFailure())
            .withHadoopJarStep(stepFactory.newScriptRunnerStep("s3://us-west-1.elasticmapreduce/libs/hue/run-hue"));
}

结果是步骤 Install Hue 成功，运行 Hue 失败。

stderr 日志：

=== Uninstalling hbase
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling security
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling search
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling sqoop
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling zookeeper
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling rdbms
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling spark
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Uninstalling impala
=== Saved registry at /var/lib/hue/app.reg
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
=== Saved /usr/lib/hue/build/env/lib/python2.6/site-packages/hue.pth
Running '/usr/lib/hue/build/env/bin/hue syncdb --noinput' with None
Running '/usr/lib/hue/build/env/bin/hue migrate --merge' with None
Traceback (most recent call last):
  File "/usr/lib/hue/build/env/bin/hue", line 9, in <module>
    load_entry_point('desktop==3.7.1-amzn-2', 'console_scripts', 'hue')()
  File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry
    execute_manager(settings)
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/__init__.py", line 459, in execute_manager
    utility.execute()
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/hue/build/env/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg/django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/usr/lib/hue/desktop/core/src/desktop/management/commands/install_all_examples.py", line 89, in handle_noargs
    raise exception
hadoop.fs.exceptions.WebHdfsException: SecurityException: Failed to obtain user group information: org.apache.hadoop.security.authorize.AuthorizationException: User: hue is not allowed to impersonate hue (error 403)
Command exiting with ret '255'

我也试过单步 运行 Hue 然后 stderr 是：

Command exiting with ret '255'

Answer 1

使用 Java sdk 安装 Hue 的正确方法是 post Install Hue 步骤作为 bootstrap 操作和 运行 Hue 作为作业流程步骤。

RunJobFlowRequest runJobFlowRequest = new RunJobFlowRequest("Main", jobFlowInstancesConfig)
            .withName("SDK_filtering" + System.currentTimeMillis())
            .withAmiVersion("3.7.0")
            .withVisibleToAllUsers(true)
            .withServiceRole("EMR_DefaultRole")
            .withJobFlowRole("EMR_EC2_DefaultRole")
            .withBootstrapActions(
                    new BootstrapActionConfig("Install Hue",
                            new ScriptBootstrapActionConfig("s3://us-west-1.elasticmapreduce/libs/hue/install-hue", null)))
            .withSteps(
                    newEnableDebugStep(),
                    newInstallHiveStep(),
                    newInstallPig(),
                    newRunHueStep()
            );

AWS EMR - 使用 Java SDK 安装 HUE

AWS EMR - install HUE using Java SDK

java

amazon-web-services

hue

elastic-map-reduce

emr