aws Sagemaker 的 AnnotationConsolidation lambda 事件的空字典

Question

我开始使用 aws sagemaker 开发我的机器学习模型，我正在尝试构建一个 lambda 函数来处理 sagemaker 标记作业的响应。我已经创建了自己的 lambda 函数，但是当我尝试读取事件内容时，我发现事件字典完全是空的，所以我没有读取任何数据。

我已经为 lambda 函数的角色赋予了足够的权限。包括： - AmazonS3FullAccess。 - AmazonSagemakerFullAccess。 - AWSLambdaBasicExecutionRole

我已经尝试将此代码用于 Post-注释 Lambda（适用于 python 3.6）：

https://docs.aws.amazon.com/sagemaker/latest/dg/sms-custom-templates-step2-demo1.html#sms-custom-templates-step2-demo1-post-annotation

以及这个 git 存储库中的这个：

https://github.com/aws-samples/aws-sagemaker-ground-truth-recipe/blob/master/aws_sagemaker_ground_truth_sample_lambda/annotation_consolidation_lambda.py

但其中 none 似乎有效。

为了创建标签作业，我使用了 boto3 的 sagemaker 函数： https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_labeling_job

这是我创建标签作业的代码：

def create_labeling_job(client,bucket_name ,labeling_job_name, manifest_uri, output_path):

    print("Creating labeling job with name: %s"%(labeling_job_name))

    response = client.create_labeling_job(
        LabelingJobName=labeling_job_name,
        LabelAttributeName='annotations',
        InputConfig={
            'DataSource': {
                'S3DataSource': {
                    'ManifestS3Uri': manifest_uri
                }
            },
            'DataAttributes': {
                'ContentClassifiers': [
                    'FreeOfAdultContent',
                ]
            }
        },
        OutputConfig={
            'S3OutputPath': output_path
        },
        RoleArn='arn:aws:myrolearn',
        LabelCategoryConfigS3Uri='s3://'+bucket_name+'/config.json',
        StoppingConditions={
            'MaxPercentageOfInputDatasetLabeled': 100,
        },
        LabelingJobAlgorithmsConfig={
            'LabelingJobAlgorithmSpecificationArn': 'arn:image-classification'
        },
        HumanTaskConfig={
            'WorkteamArn': 'arn:my-private-workforce-arn',
            'UiConfig': {
                'UiTemplateS3Uri':'s3://'+bucket_name+'/templatefile'
            },
            'PreHumanTaskLambdaArn': 'arn:aws:lambda:us-east-1:432418664414:function:PRE-BoundingBox',
            'TaskTitle': 'Title',
            'TaskDescription': 'Description',
            'NumberOfHumanWorkersPerDataObject': 1,
            'TaskTimeLimitInSeconds': 600,
            'AnnotationConsolidationConfig': {
                'AnnotationConsolidationLambdaArn': 'arn:aws:my-custom-post-annotation-lambda'
            }
        }
    )

    return response

这是我的 lambda 函数：

    print("Received event: " + json.dumps(event, indent=2))
    print("event: %s"%(event))
    print("context: %s"%(context))
    print("event headers: %s"%(event["headers"]))

    parsed_url = urlparse(event['payload']['s3Uri']);
    print("parsed_url: ",parsed_url)

    labeling_job_arn = event["labelingJobArn"]
    label_attribute_name = event["labelAttributeName"]

    label_categories = None
    if "label_categories" in event:
        label_categories = event["labelCategories"]
        print(" Label Categories are : " + label_categories)

    payload = event["payload"]
    role_arn = event["roleArn"]

    output_config = None # Output s3 location. You can choose to write your annotation to this location
    if "outputConfig" in event:
        output_config = event["outputConfig"]

    # If you specified a KMS key in your labeling job, you can use the key to write
    # consolidated_output to s3 location specified in outputConfig.
    kms_key_id = None
    if "kmsKeyId" in event:
        kms_key_id = event["kmsKeyId"]

    # Create s3 client object
    s3_client = S3Client(role_arn, kms_key_id)

    # Perform consolidation
    return do_consolidation(labeling_job_arn, payload, label_attribute_name, s3_client)

我试过使用以下方法调试事件对象：

    print("Received event: " + json.dumps(event, indent=2))

但它只打印一个空字典：Received event: {}

我希望输出类似于：

    #Content of an example event:
    {
        "version": "2018-10-16",
        "labelingJobArn": <labelingJobArn>,
        "labelCategories": [<string>],  # If you created labeling job using aws console, labelCategories will be null
        "labelAttributeName": <string>,
        "roleArn" : "string",
        "payload": {
            "s3Uri": <string>
        }
        "outputConfig":"s3://<consolidated_output configured for labeling job>"
    }

最后，当我尝试使用以下方式获取标记作业 ARN 时：

    labeling_job_arn = event["labelingJobArn"]

我刚得到一个 KeyError（这是有道理的，因为字典是空的）。

Answer 1

我发现了问题，我需要将我的 Lamda 函数使用的角色的 ARN 作为可信实体添加到用于 Sagemaker 标记作业的角色上。

我刚刚去了 Roles > MySagemakerExecutionRole > Trust Relationships 并添加了：

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::xxxxxxxxx:role/My-Lambda-Role",
           ...
        ],
        "Service": [
          "lambda.amazonaws.com",
          "sagemaker.amazonaws.com",
           ...
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

这对我有用。

Answer 2

我也在做同样的事情，但在标记对象部分我得到了失败的结果，在我的输出对象中我得到了来自 Post Lambda 函数的以下错误：

"annotation-case0-test3-metadata": {
        "retry-count": 1,
        "failure-reason": "ClientError: The JSON output from the AnnotationConsolidationLambda function could not be read. Check the output of the Lambda function and try your request again.",
        "human-annotated": "true"
    }
}

aws Sagemaker 的 AnnotationConsolidation lambda 事件的空字典

Empty dictionary on AnnotationConsolidation lambda event for aws Sagemaker

python

amazon-web-services

aws-lambda

python-3.6

amazon-sagemaker