使用 AssignmentReviewPolicy 创建 MTurk HIT 时，HIT 被错误拒绝

Question

TL;DR: 我的 AssignmentReviewPolicy 导致所有 HIT 被拒绝。 How do I change my policy to accept when "defendant" radio button is selected?

详情：

我在我的 HIT 中加入了一个黄金问题，以允许通过 AssignmentReviewPolicy 自动批准或拒绝。我只使用合格的工人，这个问题对他们来说是微不足道的。

当有人为问题 "checker" 选择值="defendant" 的单选按钮时，我的预期输出是自动批准 HIT。我正在关注 this tutorial to set the AssignmentReviewPolicy.

我的 Python，为清楚起见缩短了：

hit = client.create_hit(
    Reward='999',LifetimeInSeconds=36000, AssignmentDurationInSeconds=3600, MaxAssignments=2,
    Title='strings',
    Description='desc',
    Keywords='keywords',
    AutoApprovalDelayInSeconds=259200, #259200 sec = 3 days
    Question=question,
    AssignmentReviewPolicy={
        'PolicyName':'ScoreMyKnownAnswers/2011-09-01',
        'Parameters':[
            {'Key':'AnswerKey', 'MapEntries':[
                {'Key': 'checker', 
                 'Values':['defendant']
                }]},
            {'Key': 'ApproveIfKnownAnswerScoreIsAtLeast', 'Values':['1']},
            {'Key': 'RejectIfKnownAnswerScoreIsLessThan', 'Values':['1']},
            {'Key': 'RejectReason', 
             'Values':['Sorry, we could not approve your submission.']},
            {'Key': 'ExtendIfKnownAnswerScoreIsLessThan','Values':['1']}
        ]
    }
)

我的HTML:

<!-- Multiple Radios -->
<div class="form-group">
    <label class="col-md-4 control-label" for="checker"> What is the role of {company_name}?</label>
    <div class="col-md-4">
                <input type="radio" name="checker" value="plaintiff"> Plaintiff<br>
                <input type="radio" name="checker" value="defendant"> Defendant<br>
                <input type="radio" name="checker" value="judge"> Judge<br>
                <input type="radio" name="checker" value="attorney"> Attorney<br>
    </div>
</div>

意想不到的行为是无论选择哪个选项，包括"defendant"，答案都被拒绝

我尝试使用 client.list_review_policy_results_for_hit(HITId='')，但我不确定要从输出中去掉什么：

{'HITId': '',
 'AssignmentReviewPolicy': {'PolicyName': 'ScoreMyKnownAnswers/2011-09-01'},
 'AssignmentReviewReport': {'ReviewResults': [{'ActionId': '3DDXUM5F3Q2LH7RXIDK51QITDWP9KO',
    'SubjectId': '',
    'SubjectType': 'Assignment',
    'QuestionId': '',
    'Key': 'KnownAnswerScore',
    'Value': '0'},
   {'ActionId': '3DDXUM5F3Q2LH7RXIDK51QITDWP9KO',
    'SubjectId': '',
    'SubjectType': 'Assignment',
    'QuestionId': 'checker',
    'Key': 'AgreedWithKnownAnswer',
    'Value': 'false'}],
  'ReviewActions': [{'ActionId': '3DDXUM5F3Q2LH7RXIDK51QITDWP9KO',
    'ActionName': 'review',
    'TargetId': '',
    'TargetType': 'Assignment',
    'Status': 'Succeeded',
    'Result': 'Assignment 3VBEN272OTK0OGFSGT KnownAnswerScore is 0%. Assignment rejected as a result. HIT extended as a result.'},
   {'ActionId': '38MQYK43N4YBSE3EAWAYNKWE10LP2D',
    'ActionName': 'reject',
    'TargetId': '3VBEN272OTK0OGFSGT',
    'TargetType': 'Assignment',
    'Status': 'Succeeded'},
   {'ActionId': '3SBIQ36PU450DWZXEWO36U167ELP3O',
    'ActionName': 'extend',
    'TargetId': '3QX22DUVPRJEBK749RF6N1SOE16VME',
    'TargetType': 'HIT',
    'Status': 'Succeeded',
    'Result': 'Extended by 1 assignment'}]},
 'NextToken': 'p1:HpeunOuRoWcvUyy1vi8zwEVMVptX3tv5BkWSag==',
 'ResponseMetadata': {'RequestId': '2f89af041eece',
  'HTTPStatusCode': 200,
  'HTTPHeaders': {'x-amzn-requestid': '2f89af0b-667dc841eece',
   'content-type': 'application/x-amz-json-1.1',
   'content-length': '1260',
   'date': 'Tue, 24 Sep 2019 20:23:24 GMT'},
  'RetryAttempts': 2}}

Answer 1

问题在于答案中单选按钮 return 的格式。本教程对此表示不正确。

返回的实际答案格式为：

OrderedDict([('QuestionIdentifier',
              'checker.defendant'),
             ('FreeText', 'true')]),

意思是正确的 AnswerKey 应该是：

{'Key': 'AnswerKey', 'MapEntries': [
    {'Key': 'checker.defendant',
     'Values': ['true']
     }]},

使用 AssignmentReviewPolicy 创建 MTurk HIT 时，HIT 被错误拒绝

When creating an MTurk HIT with AssignmentReviewPolicy, HITs are rejected incorrectly

python

mechanicalturk

amazon-web-services

boto3