mlr：用于特征选择的嵌套重采样

Question

我正在运行进行基准实验，我在其中调整过滤学习器，类似于嵌套重采样下的 mlr 教程中标题为 "Example 3: One task, two learners, feature filtering with tuning" 的示例。我的代码如下：

library(survival)
library(mlr)

data(veteran)
set.seed(24601)
configureMlr(show.learner.output=TRUE, show.info=TRUE)

task_id = "MAS"
mas.task <- makeSurvTask(id = task_id, data = veteran, target = c("time", "status"))
mas.task <- createDummyFeatures(mas.task)

inner = makeResampleDesc("CV", iters=2, stratify=TRUE)  # Tuning
outer = makeResampleDesc("CV", iters=3, stratify=TRUE)  # Benchmarking

cox.lrn <- makeLearner(cl="surv.coxph", id = "coxph", predict.type="response")
cox.filt.uni.thresh.lrn = makeTuneWrapper(
  makeFilterWrapper(
    makeLearner(cl="surv.coxph", id = "cox.filt.uni.thresh", predict.type="response"), 
    fw.method="univariate.model.score", 
    perf.learner=cox.lrn
  ), 
  resampling = inner, 
  par.set = makeParamSet(makeDiscreteParam("fw.threshold", values=c(0.5, 0.6, 0.7))), 
  control = makeTuneControlGrid(),
  show.info = TRUE)

learners = list( cox.filt.uni.thresh.lrn )  
bmr = benchmark(learners=learners, tasks=mas.task, resamplings=outer, measures=list(cindex), show.info = TRUE)

使用这种方法，似乎外部重采样循环的每次迭代都会为 fw.threshold 使用可能不同的值 - 它将使用在内部循环中确定的最佳值。我的问题是，这是可以接受的还是首先使用 tuneParams 和交叉验证调整该参数会更好，然后运行使用先前调整的参数进行基准测试，如下所示：

library(survival)
library(mlr)

data(veteran)
set.seed(24601)
configureMlr(show.learner.output=TRUE, show.info=TRUE)

task_id = "MAS"
mas.task <- makeSurvTask(id = task_id, data = veteran, target = c("time", "status"))
mas.task <- createDummyFeatures(mas.task)

inner = makeResampleDesc("CV", iters=2, stratify=TRUE)  # Tuning
outer = makeResampleDesc("CV", iters=3, stratify=TRUE)  # Benchmarking

cox.lrn <- makeLearner(cl="surv.coxph", id = "coxph", predict.type="response")
cox.filt.uni.thresh.lrn = 
  makeFilterWrapper(
    makeLearner(cl="surv.coxph", id = "cox.filt.uni.thresh", predict.type="response"), 
    fw.method="univariate.model.score", 
    perf.learner=cox.lrn
  )
params = makeParamSet(makeDiscreteParam("fw.threshold", values=c(0.5, 0.6, 0.7)))
ctrl = makeTuneControlGrid()

tuned.params = tuneParams(cox.filt.uni.thresh.lrn, mas.task, resampling = inner, par.set=params, control=ctrl, show.info = TRUE)
tuned.lrn = setHyperPars(cox.filt.uni.thresh.lrn, par.vals = tuned.params$x)

learners = list( tuned.lrn )  
bmr = benchmark(learners=learners, tasks=mas.task, resamplings=outer, measures=list(cindex), show.info = TRUE)

在这种情况下，第二种方法给出的结果稍差，但我想知道哪种方法是正确的。

Answer 1

Using this method it seems that each iteration of the outer resampling loop will use a possibly different value for fw.threshold - it will use the value that was determined best in the inner loop. My question is, is this acceptable or would it be better to tune that parameter first, using tuneParams and cross validation, and then run the benchmark using the previously tuned parameters like this:

不，因为在这种情况下您的模型已经看到了数据。这引入了偏见。您应该在 CV 中对每个折叠分别进行所有优化。所以方法#1 是正确的。

此外，我建议不要使用网格搜索，而是使用随机搜索或 MBO。

mlr：用于特征选择的嵌套重采样

mlr: nested resampling for feature selection

r

mlr