在 mlR 中计算 AUC 留一法交叉验证？

Question

这是一个简短的问题，只是为了确保我没有以愚蠢的方式这样做。我想使用 auc 作为我在 mlr 中的度量，并且由于样本量小，我也使用 LOO。当然，在LOO交叉验证方案中，测试样本总是只有一个实例，所以无法计算auc。当然，我们可以事后计算它，查看预测，当我们想将它用作嵌套交叉验证的内循环中的度量时，问题就会发生。像这样（你必须自己定义 binaryTask）：

require(mlr)    
#for example purposes we will decide which one is better, vanilla LDA or
#vanilla SVM, in the task specified below
bls = list(makeLearner("classif.lda"),makeLearner("classif.svm"))
#modelMultiplexer allows us to search whole parameter spaces between models
#as if the models themselves were parameters
lrn = makeModelMultiplexer(bls)
#to calculate AUC we need some continuous output, so we set 
#predictType to probabilities
lrn = setPredictType(lrn, "prob")
lrn = setId(lrn, "Model Multiplexer")
#here we could pass the parameters to be tested to both SVM and LDA,
#let's not pass anything so we test the vanilla classifiers instead
ps = makeModelMultiplexerParamSet(lrn)
#finally, the resample strategy, Leave-One-Out ("LOO") in our case
rdesc = makeResampleDesc("LOO")
#parameter space search strategy, in our case we only have one parameter:
#the model. So, a simple grid search will do the trick
ctrl = makeTuneControlGrid()
#The inner CV loop where we choose the best model in the validation data
tune = makeTuneWrapper(lrn, rdesc, par.set = ps, control = ctrl, measure = auc, show.info = FALSE) 
#The outer CV loop where we obtain the performace of the selected model 
#in the test data. mlR is a great interface, we could have passed a list 
#of classifiers and tasks here instead and do it all in one go 
#(beware your memory limitation!)
res = benchmark(tune, binaryTask, rdesc, measure = auc)

您不能在两个循环中都这样使用 auc。我们如何才能mlr评估所有测试样本的度量而不是每次都重新采样？

Answer 1

您可以对内循环使用不同的重采样策略，然后使用 auc:

library(mlr)    

ps = makeParamSet(
  makeNumericLearnerParam(id = "cp", default = 0.01, lower = 0, upper = 1)
)
ctrl = makeTuneControlRandom(maxit = 10)
inner = makeResampleDesc("Subsample")
lrn = makeLearner("classif.rpart", predict.type = "prob")
tune = makeTuneWrapper(lrn, resampling = inner, par.set = ps, control = ctrl, measure = auc)

outer = makeResampleDesc("LOO")
r = resample(tune, bc.task, resampling = outer, extract = getTuneResult, measure = auc)

您也可以只获取重采样结果并对其计算任意性能度量，例如performance(r$pred, auc).

在 mlR 中计算 AUC 留一法交叉验证？

Calculating AUC Leave-One-Out cross validation in mlR?

r

classification

cross-validation

auc

mlr