我的 EMR 集群在哪里

Question

我正在尝试在 java 上创建 EMR 集群，但我在 EMR 集群列表中找不到它，也看不到在 EC2 上请求的实例。

确实存在 EMR 角色：

sqlInjection@VirtualBox:~$ aws iam list-roles | grep EMR
            "RoleName": "EMR_DefaultRole", 
            "Arn": "arn:aws:iam::removed:role/EMR_DefaultRole"
            "RoleName": "EMR_EC2_DefaultRole", 
            "Arn": "arn:aws:iam::removed:role/EMR_EC2_DefaultRole"

现在我的 java 代码：

AWSCredentials awsCredentials = new BasicAWSCredentials(awsKey, awsKeySecret);
AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials);

StepFactory stepFactory = new StepFactory();
        StepConfig enabledebugging = new StepConfig()
               .withName("Enable debugging")
               .withActionOnFailure("TERMINATE_JOB_FLOW")
               .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

    HadoopJarStepConfig hadoopConfig1 = new HadoopJarStepConfig()
    .withJar("s3://foo.bucket/hadoop_jar/2015-01-12/foo.jar")
    .withMainClass("com.strackoverflow.DriverFoo") // optional main class, this can be omitted if jar above has a manifest
    .withArgs("--input=s3://foo.bucket/logs/,s3://foo.bucket/morelogs/", "--output=s3://foo.bucket/myEMROutput" , "--inputType=text"); // i have custom java code to handle the --input, --output and --inputType parameters


    StepConfig customStep = new StepConfig("Step1", hadoopConfig1);

    Collection <StepConfig> steps = new ArrayList<StepConfig>();
    {
        steps.add(enabledebugging);
        steps.add(customStep);
    }


    JobFlowInstancesConfig instancesConfig = new JobFlowInstancesConfig()
    .withEc2KeyName("fookey") //not fookey.pem
    .withInstanceCount(2)
    .withKeepJobFlowAliveWhenNoSteps(false) // on aws example is set to true
    .withMasterInstanceType("m1.medium")
    .withSlaveInstanceType("m1.medium");


    RunJobFlowRequest request = new RunJobFlowRequest()
     .withName("java programatic request")
     .withAmiVersion("3.3.1")
     .withSteps(steps) // on the amazon example is lunched debug and hive, here is debug and a jar
     .withLogUri("s3://devel.rui/emr_clusters/pr01/")
     .withInstances(instancesConfig)
     .withVisibleToAllUsers(true);

       RunJobFlowResult result = emr.runJobFlow(request);

       System.out.println("toString "+ result.toString());
       System.out.println("getJobFlowId "+ result.getJobFlowId());
       System.out.println("hashCode "+ result.hashCode());

我的集群在哪里？我在集群列表中看不到它，未创建输出文件夹，日志文件夹保持为空并且在 EC2 上看不到任何实例。

由程序输出这个

toString {JobFlowId: j-2xxxxxxU}
getJobFlowId j-2xxxxxU
hashCode -1xxxxx4

我已按照此处的说明创建集群 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/calling-emr-with-java-sdk.html

然后创建 java 作业 http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-common-programming-sample.html

Answer 1

在亚马逊示例中，未配置区域。

配置区域后集群正常启动。

    AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(awsCredentials);
    emr.setRegion(Region.getRegion(Regions.EU_WEST_1));

我的 EMR 集群在哪里

where is my EMR cluster

java

amazon-web-services

emr