h2o AutoML 结果令人困惑

Question

我在我的 java 代码中使用 AutoML，我使用 AUC 作为我的 sort_metric。经过 20 分钟的训练后，我得到了以下训练总结：

09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.869 Info  ModelTraining    Built: 1 models for search: DeepLearning hyperparameter search 1
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.869 Info  ModelTraining    New leader: DeepLearning_grid_0_AutoML_20180901_092939_model_0, AUC: 0.9170303558590623
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.869 Info  ModelTraining    DeepLearning hyperparameter search 1 complete
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.869 Info  ModelTraining    AutoML: out of time; skipping DL hyperparameter search
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.870 Info  ModelTraining    AutoML: out of time; skipping DL hyperparameter search
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.870 Info  ModelTraining    StackedEnsemble builds skipped due to the exclude_algos option.
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 09:51:05.870 Info  Workflow         AutoML: build done; built 2 models
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: Leaderboard for project my.proj (models sorted in order of AUC, best first):
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: #                                            model_id       auc   logloss  mean_per_class_error      rmse       mse
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 0  DeepLearning_grid_0_AutoML_20180901_092939_model_0  0.917030  0.273277              0.147665  0.189239  0.035812
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 1  DeepLearning_0_AutoML_20180901_092939               0.937039  0.150729              0.214383  0.193596  0.037479
09-01 09:51:05.872 127.0.0.1:54321       18273  FJ-1-9    INFO: 2  DeepLearning_0_AutoML_20180901_092936               0.958391  0.127028              0.181791  0.179389  0.032180

似乎"DeepLearning_0_AutoML_20180901_092936"是最好的，因为它的AUC值最高，但实际上"DeepLearning_grid_0_AutoML_20180901_092939_model_0"是第一个，并且 AtuoML.leader() 也 return 模型 "DeepLearning_grid_0_AutoML_20180901_092939_model_0"。那么哪个最好？

以下是运行 AutoML 的代码：

    AutoMLBuildSpec autoMLBuildSpec = new AutoMLBuildSpec();

    autoMLBuildSpec.input_spec.training_frame = frame._key;
    autoMLBuildSpec.input_spec.response_column = "class";
    autoMLBuildSpec.input_spec.sort_metric = "AUC";

    autoMLBuildSpec.build_control.balance_classes = true;
    autoMLBuildSpec.build_control.class_sampling_factors = new float[2];
    autoMLBuildSpec.build_control.class_sampling_factors[0] = 1.0f;
    autoMLBuildSpec.build_control.class_sampling_factors[1] = 1.0f;
    autoMLBuildSpec.build_control.nfolds = nfolds;
    autoMLBuildSpec.build_control.keep_cross_validation_models = true;
    autoMLBuildSpec.build_control.keep_cross_validation_predictions = true;
    autoMLBuildSpec.build_control.project_name = "my.proj";
    HyperSpaceSearchCriteria.RandomDiscreteValueSearchCriteria randomDiscreteValueSearchCriteria = new HyperSpaceSearchCriteria.RandomDiscreteValueSearchCriteria();
    randomDiscreteValueSearchCriteria.set_max_runtime_secs(Double.parseDouble(autoModelRuntimeSeconds));
    randomDiscreteValueSearchCriteria.set_stopping_metric(ScoreKeeper.StoppingMetric.AUTO);
    randomDiscreteValueSearchCriteria.set_stopping_tolerance(0.0);
    autoMLBuildSpec.build_control.stopping_criteria = randomDiscreteValueSearchCriteria;

    AutoMLBuildSpec.AutoMLBuildModels autoMLBuildModels = new AutoMLBuildSpec.AutoMLBuildModels();
    autoMLBuildModels.exclude_algos = new AutoML.algo[4];
    autoMLBuildModels.exclude_algos[0] = AutoML.algo.DRF;
    autoMLBuildModels.exclude_algos[1] = AutoML.algo.GBM;
    autoMLBuildModels.exclude_algos[2] = AutoML.algo.GLM;
    autoMLBuildModels.exclude_algos[3] = AutoML.algo.StackedEnsemble;

    autoMLBuildSpec.build_models = autoMLBuildModels;

    logger.info("begin training ...");
    AutoML aml = AutoML.makeAutoML(Key.make(), new Date(), autoMLBuildSpec);
    AutoML.startAutoML(aml);
    AutoML.startAutoML(autoMLBuildSpec).get();
    logger.info("training finished.");

    for (Model model: aml.leaderboard().getModels()) {
            logger.info("========================================================================================================");
            logger.info("model key: {}", model._key);
            logger.info("_scoring_history");
            logger.info(model._output._scoring_history.toString(10, true));
            logger.info("model auc: {}", Utils.doubleToString(model.auc(), 6));
            logger.info("========================================================================================================");
    }
    logger.info("leader model scoring history:");
    logger.info(aml.leader()._output._scoring_history.toString(10, true));

Answer 1

请尝试将指标指定为小写，以便在使用 "auc" 时排行榜按降序排序（我们似乎并没有强制执行此案例，我已经提交了 JIRA 解决这个问题的票）：

autoMLBuildSpec.input_spec.sort_metric = "auc";

另外仅供参考，您不需要 "start" automl 2 次。基本上

AutoML aml = AutoML.makeAutoML(Key.make(), new Date(), autoMLBuildSpec);
AutoML.startAutoML(aml);
AutoML.startAutoML(autoMLBuildSpec).get();

可以换成

AutoML.startAutoML(autoMLBuildSpec).get();

希望对您有所帮助！

h2o AutoML 结果令人困惑

h2o AutoML result is confusing

metrics

h2o

automl