延长 HIT 直到达成协议 MTurkR

Question

我正在使用 MTurkR 到 post 到 mTurk 的 HIT，我无法扩展 HIT，直到工人之间达成共识（或直到共有 5 个 HIT post编）。

当输入两个不同的响应时，HIT 不会延长。我的代码来自 MTurkR documentation pg。 45.我的代码如下：

# ##############################################
# SET PARAMTERS FOR HITS
# ##############################################
layout="XXXXXXXXXXX"
    #format for sandbox question. Get this from MTURK site
annotation.v="Question1"
assignments.v="2"
title.v="TITLE"
description.v="DESCRIPTION."
reward.v=".00"
duration.v=seconds(hour=1)
expiration.v=seconds(days=4)
keywords.v="survey"
auto.approval.delay.v=seconds(days=1)


# ##############################################
# EXTEND HIT UNTIL AGREEMENT
# ##############################################
TurkAgreement=list(QuestionIds=c("Question1"),
  QuestionAgreementThreshold=49, #at least 50% agree
  ExtendIfHITAgreementScoreIsLessThan=50,
  ExtendMinimumTimeInSeconds=3600,
  ExtendMaximumAssignments=5,
  DisregardAssignmentIfRejected=TRUE)

policya=do.call(GenerateHITReviewPolicy,TurkAgreement)


# ##############################################
# CREATE HITS
# ##############################################
hits=NULL
for(i in 1:length(DF)){
  hits.i=CreateHIT(
    hitlayoutid=layout,
    hitlayoutparameters=GenerateHITLayoutParameter(c("XX","XX","XX"), c(DF[i,1],DF[i,2],DF[i,3])),
    annotation=annotation.v[i],
    assignments=assignments.v,
    title=title.v,
    description=description.v,
    reward=reward.v,
    duration=duration.v,
    expiration=expiration.v,
    keywords=keywords.v,
    auto.approval.delay=auto.approval.delay.v,
    qual.req=qualReqs,
    hit.review.policy=policya,
    sandbox=sandbox.v)
    hits=rbind(hits,hits.i)}

代码生成 2 个 HIT（由 assignments.v 指定）但 HIT 没有扩展。

我的会话信息如下：

> sessionInfo()
R version 3.1.1 (2014-07-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] MTurkR_0.6

loaded via a namespace (and not attached):
[1] digest_0.6.4   RCurl_1.95-4.3 tcltk_3.1.1    XML_3.98-1.1

Answer 1

首先是关于您发布的代码的几个要点：

因为您要添加 hit.review.policy，所以不需要 assignments 参数。仅此语句将立即填充两个 HIT。
在定义 hit.review.policy 的术语之前，请先查看 Amazon's documentation。请特别注意 QuestionAgreementThreshold 的值。因为大于此值的值被认为是一致的答案，所以您的原始代码指示大于 49% 的值是一致的答案。还要检查 ExtendIfHITAgreementScoreIsLessThan 这个值可能应该是 QuestionAgreementThreshold 值的 +1，以便捕获所有可能的协议值。
最后，确保 HIT 响应的答案值与 QuestionIds 对象相对应。当我查看我的 HIT 的答案时，答案被称为 "question"，所以我在这里替换了这个值。

综上所述，这是现在正确扩展 HIT 的修订代码：

# ##############################################
# SET PARAMTERS FOR HITS
# ##############################################
layout="XXXXXXXXXXX"
    #format for sandbox question. Get this from MTURK site
annotation.v="question" #NOTE CHANGE HERE
#assignments.v="2"  #NOTE CHANGE HERE (COMMENTED THIS OUT)
title.v="TITLE"
description.v="DESCRIPTION."
reward.v=".00"
duration.v=seconds(hour=1)
expiration.v=seconds(days=4)
keywords.v="survey"
auto.approval.delay.v=seconds(days=1)


# ##############################################
# EXTEND HIT UNTIL AGREEMENT
# ##############################################
TurkAgreement=list(QuestionIds=c("question"), #NOTE CHANGE HERE
  QuestionAgreementThreshold=50, #at least 50% agree  #NOTE CHANGE HERE
  ExtendIfHITAgreementScoreIsLessThan=51, #NOTE CHANGE HERE
  ExtendMinimumTimeInSeconds=3600,
  ExtendMaximumAssignments=5,
  DisregardAssignmentIfRejected=TRUE)

policya=do.call(GenerateHITReviewPolicy,TurkAgreement)


# ##############################################
# CREATE HITS
# ##############################################
hits=NULL
for(i in 1:length(DF)){
  hits.i=CreateHIT(
    hitlayoutid=layout,
    hitlayoutparameters=GenerateHITLayoutParameter(c("XX","XX","XX"), c(DF[i,1],DF[i,2],DF[i,3])),
    annotation=annotation.v[i],
    assignments=assignments.v,
    title=title.v,
    description=description.v,
    reward=reward.v,
    duration=duration.v,
    expiration=expiration.v,
    keywords=keywords.v,
    auto.approval.delay=auto.approval.delay.v,
    qual.req=qualReqs,
    hit.review.policy=policya,
    sandbox=sandbox.v)
    hits=rbind(hits,hits.i)}

另请注意 MTurkR Package 页面上的文档。 44（结束）令人困惑。该文档提供了以下示例：

lista<-list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 49, # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 50,
...

但参数 QuestionAgreementThreshold 实际上指定了至少 49% 的一致性（即，如果 2 个土耳其人拆分答案，HIT 将不会延长）。除非是有意为之，否则最好使用以下代码：

lista<-list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 50, #NOTE CHANGE HERE # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 51, #NOTE CHANGE HERE
...

延长 HIT 直到达成协议 MTurkR

Extend HITs until agreement MTurkR

r

mechanicalturk