Ruby 的 EMR 流作业在 IAM 策略更改后失败

Question

最近亚马逊更改了其 EMR 工作政策以定义 IAM 角色。我得到了最新的 awssdk.dll 并升级了代码。我创建了所需的策略并成功获得了我的旧猪工作运行。

我的 Ruby 作业不工作。我一直收到找不到映射器文件的错误，即使它在 S3 上。我已经验证映射器文件是 public- 读给所有人并且路径是 correct/exists.

感谢任何帮助..谢谢

Caused by: java.lang.RuntimeException: configuration exception at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:232) at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22 more Caused by: java.io.IOException: Cannot run program "s3://er-hadoop/Scripts/Audit/Debugging/UserId_Picker/WebPageAudit_GetDataForSpecifiedUsers_Mapper.rb": error=2, No such file or directory

StreamingStep stepConfig = new StreamingStep { Inputs = new List<string> { "s3://er-upload-filestore-east/WebPageAudit/2015/04/01/*.gz", }, Output = "s3://er-hadoop/Output/Mangal/testawschanges", Mapper = "s3://er-hadoop/Scripts/Audit/Debugging/UserId_Picker/WebPageAudit_GetDataForSpecifiedUsers_Mapper.rb", Reducer = "s3://er-hadoop/Scripts/Audit/Debugging/UserId_Picker/WebPageAudit_GetDataForSpecifiedUsers_Reducer.rb", }; stepConfig.AddHadoopConfig("-jobconf", "mapred.output.compress=true"); HadoopJarStepConfig config = stepConfig.ToHadoopJarStepConfig();

Answer 1

        const string mapper_reducer_Path = "FULL S3 Path ";
        const string mapperFileName = "YourMapper.rb";
        const string reducerFileName = "YourReducer.rb";

        string mapperFilePath = string.Format("{0}/{1}", mapper_reducer_Path, mapperFileName);
        string reducerFilePath = string.Format("{0}/{1}", mapper_reducer_Path, reducerFileName);

        StreamingStep stepConfig = new StreamingStep
        {
            Inputs = new List<string>
            {
                "YOURInput",
            },
            Output = "YOUrOUTPUT",
            Mapper = mapperFileName,
            Reducer = reducerFileName,
        };

         HadoopJarStepConfig config = stepConfig.ToHadoopJarStepConfig();
         config.Args.Insert(0, "-files");
        config.Args.Insert(1, string.Format("{0},{1}", mapperFilePath, reducerFilePath));

Ruby 的 EMR 流作业在 IAM 策略更改后失败

EMR streaming jobw with Ruby fails after IAM policy changes

amazon-emr