Python 中带有自定义评估函数的 xgboost 的意外行为
Unexpected behavior from xgboost in Python with Custom Evaluation Function
我正在使用带有自定义评估函数的 xgboost，我想实施提前停止设置 150 轮的限制。
我得到了 4 个评估指标，而不是预期的 2 个，我不知道如何解释它们。此外，我也不确定如何激活提前停止设置限制（例如，150 发）。
一个可重现的例子：
import numpy as np
def F1_eval_gen(preds, labels):
t = np.arange(0, 1, 0.005)
f = np.repeat(0, 200)
results = np.vstack([t, f]).T
# assuming labels only containing 0's and 1's
n_pos_examples = sum(labels)
if n_pos_examples == 0:
n_pos_examples = 1
for i in range(200):
pred_indexes = (preds >= results[i, 0])
TP = sum(labels[pred_indexes])
FP = len(labels[pred_indexes]) - TP
precision = 0
recall = TP / n_pos_examples
if (FP + TP) > 0:
precision = TP / (FP + TP)
if (precision + recall > 0):
F1 = 2 * precision * recall / (precision + recall)
else:
F1 = 0
results[i, 1] = F1
return (max(results[:, 1]))
def F1_eval(preds, dtrain):
res = F1_eval_gen(preds, dtrain.get_label())
return 'f1_err', 1-res
from sklearn import datasets
from sklearn.model_selection import *
skl_data = datasets.load_breast_cancer()
X = skl_data.data
y = skl_data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
scale_pos_weight = sum(y_train == 0)/sum(y_train == 1)
base_score = sum(y_train == 1)/len(y_train)
max_depth = 6
learning_rate = 0.1
gamma = 0
min_child_weight = 1
subsample = 0.8
colsample_bytree = 0.8
colsample_bylevel = 1
reg_alpha = 0
reg_lambda = 1
clf = xgb.XGBClassifier(max_depth= max_depth, learning_rate= learning_rate,silent=False, objective='binary:logistic', \
booster='gbtree', n_jobs=8, nthread=None, gamma=gamma, min_child_weight=min_child_weight, max_delta_step=0, \
subsample= subsample, colsample_bytree=colsample_bytree, colsample_bylevel=colsample_bylevel, \
reg_alpha= reg_alpha, reg_lambda=1, scale_pos_weight= scale_pos_weight, base_score= base_score)
clf.fit(X_train, y_train,
eval_set=[(X_train, y_train), (X_test, y_test)], eval_metric= F1_eval, verbose=True)
..................
[94] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[95] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[96] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[97] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[98] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[99] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
clf = xgb.XGBClassifier(max_depth= max_depth, niterations = 1000, learning_rate= learning_rate,silent=False, \
objective='binary:logistic', booster='gbtree', n_jobs=8, nthread=None, gamma=gamma,\
min_child_weight=min_child_weight, max_delta_step=0, \
subsample= subsample, colsample_bytree=colsample_bytree, colsample_bylevel=colsample_bylevel, \
reg_alpha= reg_alpha, reg_lambda=1, scale_pos_weight= scale_pos_weight, base_score= base_score)
clf.fit(X_train, y_train, early_stopping_rounds= 25,
eval_set=[(X_train, y_train), (X_test, y_test)], eval_metric= F1_eval, verbose=True)
[0] validation_0-error:0.386813 validation_1-error:0.315789 validation_0-f1_err:0.032609 validation_1-f1_err:0.031847
Multiple eval metrics have been passed: 'validation_1-f1_err' will be used for early stopping.
Will train until validation_1-f1_err hasn't improved in 25 rounds.
[1] validation_0-error:0.131868 validation_1-error:0.078947 validation_0-f1_err:0.016216 validation_1-f1_err:0.031056
[2] validation_0-error:0.048352 validation_1-error:0.052632 validation_0-f1_err:0.012522 validation_1-f1_err:0.037037
[3] validation_0-error:0.032967 validation_1-error:0.04386 validation_0-f1_err:0.008977 validation_1-f1_err:0.031447
[4] validation_0-error:0.01978 validation_1-error:0.04386 validation_0-f1_err:0.010753 validation_1-f1_err:0.031447
[5] validation_0-error:0.015385 validation_1-error:0.035088 validation_0-f1_err:0.008977 validation_1-f1_err:0.025316
[6] validation_0-error:0.013187 validation_1-error:0.04386 validation_0-f1_err:0.010676 validation_1-f1_err:0.025316
[7] validation_0-error:0.017582 validation_1-error:0.04386 validation_0-f1_err:0.010638 validation_1-f1_err:0.018868
[8] validation_0-error:0.013187 validation_1-error:0.04386 validation_0-f1_err:0.008913 validation_1-f1_err:0.025
[9] validation_0-error:0.008791 validation_1-error:0.04386 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[10] validation_0-error:0.010989 validation_1-error:0.04386 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[11] validation_0-error:0.008791 validation_1-error:0.04386 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[12] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[13] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.007117 validation_1-f1_err:0.025
[14] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[15] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[16] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[17] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[18] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[19] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[20] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[21] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[22] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[23] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[24] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[25] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[26] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[27] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003584 validation_1-f1_err:0.018868
[28] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003584 validation_1-f1_err:0.018868
[29] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[30] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.001789 validation_1-f1_err:0.018868
[31] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.001789 validation_1-f1_err:0.018868
[32] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.001789 validation_1-f1_err:0.018868
Stopping. Best iteration:
[7] validation_0-error:0.017582 validation_1-error:0.04386 validation_0-f1_err:0.010638 validation_1-f1_err:0.018868
XGBClassifier(base_score=0.6131868131868132, booster='gbtree',
colsample_bylevel=1, colsample_bytree=0.8, gamma=0,
learning_rate=0.1, max_delta_step=0, max_depth=6,
min_child_weight=1, missing=None, n_estimators=100, n_jobs=8,
niterations=1000, nthread=None, objective='binary:logistic',
random_state=0, reg_alpha=0, reg_lambda=1,
scale_pos_weight=0.6308243727598566, seed=None, silent=False,
subsample=0.8)
你得到 4 个评估矩阵，因为 xgboost 以某种方式向你的 eval_set 添加了另一个评估指标。就个人而言，我使用的是核心 xgboost 而不是 scikit warp up。因此，如果您想了解更多信息，请阅读文档。
对于early_stopping，您必须将n_estimators=1000（或您想要的迭代次数）设置为xgb.XGBClassifier
中的参数
并在 clf.fit 中设置 early_stopping_rounds=50（或您想要的任何值）。
Here's the documentation。
快速了解提前停止的作用
提前停止决定何时需要停止增强算法以避免过度拟合。它通过评估您在 eval_set 中定义的 tuple (X_test, y_test) 来实现。如果评估错误在 50 次迭代中没有减少，early_stopping 将停止提升。
我正在使用带有自定义评估函数的 xgboost，我想实施提前停止设置 150 轮的限制。
我得到了 4 个评估指标，而不是预期的 2 个，我不知道如何解释它们。此外，我也不确定如何激活提前停止设置限制（例如，150 发）。
一个可重现的例子：
import numpy as np
def F1_eval_gen(preds, labels):
t = np.arange(0, 1, 0.005)
f = np.repeat(0, 200)
results = np.vstack([t, f]).T
# assuming labels only containing 0's and 1's
n_pos_examples = sum(labels)
if n_pos_examples == 0:
n_pos_examples = 1
for i in range(200):
pred_indexes = (preds >= results[i, 0])
TP = sum(labels[pred_indexes])
FP = len(labels[pred_indexes]) - TP
precision = 0
recall = TP / n_pos_examples
if (FP + TP) > 0:
precision = TP / (FP + TP)
if (precision + recall > 0):
F1 = 2 * precision * recall / (precision + recall)
else:
F1 = 0
results[i, 1] = F1
return (max(results[:, 1]))
def F1_eval(preds, dtrain):
res = F1_eval_gen(preds, dtrain.get_label())
return 'f1_err', 1-res
from sklearn import datasets
from sklearn.model_selection import *
skl_data = datasets.load_breast_cancer()
X = skl_data.data
y = skl_data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)
scale_pos_weight = sum(y_train == 0)/sum(y_train == 1)
base_score = sum(y_train == 1)/len(y_train)
max_depth = 6
learning_rate = 0.1
gamma = 0
min_child_weight = 1
subsample = 0.8
colsample_bytree = 0.8
colsample_bylevel = 1
reg_alpha = 0
reg_lambda = 1
clf = xgb.XGBClassifier(max_depth= max_depth, learning_rate= learning_rate,silent=False, objective='binary:logistic', \
booster='gbtree', n_jobs=8, nthread=None, gamma=gamma, min_child_weight=min_child_weight, max_delta_step=0, \
subsample= subsample, colsample_bytree=colsample_bytree, colsample_bylevel=colsample_bylevel, \
reg_alpha= reg_alpha, reg_lambda=1, scale_pos_weight= scale_pos_weight, base_score= base_score)
clf.fit(X_train, y_train,
eval_set=[(X_train, y_train), (X_test, y_test)], eval_metric= F1_eval, verbose=True)
..................
[94] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[95] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[96] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[97] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[98] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
[99] validation_0-error:0 validation_1-error:0.035088 validation_0-f1_err:0 validation_1-f1_err:0.018634
clf = xgb.XGBClassifier(max_depth= max_depth, niterations = 1000, learning_rate= learning_rate,silent=False, \
objective='binary:logistic', booster='gbtree', n_jobs=8, nthread=None, gamma=gamma,\
min_child_weight=min_child_weight, max_delta_step=0, \
subsample= subsample, colsample_bytree=colsample_bytree, colsample_bylevel=colsample_bylevel, \
reg_alpha= reg_alpha, reg_lambda=1, scale_pos_weight= scale_pos_weight, base_score= base_score)
clf.fit(X_train, y_train, early_stopping_rounds= 25,
eval_set=[(X_train, y_train), (X_test, y_test)], eval_metric= F1_eval, verbose=True)
[0] validation_0-error:0.386813 validation_1-error:0.315789 validation_0-f1_err:0.032609 validation_1-f1_err:0.031847
Multiple eval metrics have been passed: 'validation_1-f1_err' will be used for early stopping.
Will train until validation_1-f1_err hasn't improved in 25 rounds.
[1] validation_0-error:0.131868 validation_1-error:0.078947 validation_0-f1_err:0.016216 validation_1-f1_err:0.031056
[2] validation_0-error:0.048352 validation_1-error:0.052632 validation_0-f1_err:0.012522 validation_1-f1_err:0.037037
[3] validation_0-error:0.032967 validation_1-error:0.04386 validation_0-f1_err:0.008977 validation_1-f1_err:0.031447
[4] validation_0-error:0.01978 validation_1-error:0.04386 validation_0-f1_err:0.010753 validation_1-f1_err:0.031447
[5] validation_0-error:0.015385 validation_1-error:0.035088 validation_0-f1_err:0.008977 validation_1-f1_err:0.025316
[6] validation_0-error:0.013187 validation_1-error:0.04386 validation_0-f1_err:0.010676 validation_1-f1_err:0.025316
[7] validation_0-error:0.017582 validation_1-error:0.04386 validation_0-f1_err:0.010638 validation_1-f1_err:0.018868
[8] validation_0-error:0.013187 validation_1-error:0.04386 validation_0-f1_err:0.008913 validation_1-f1_err:0.025
[9] validation_0-error:0.008791 validation_1-error:0.04386 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[10] validation_0-error:0.010989 validation_1-error:0.04386 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[11] validation_0-error:0.008791 validation_1-error:0.04386 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[12] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.007143 validation_1-f1_err:0.025
[13] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.007117 validation_1-f1_err:0.025
[14] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[15] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[16] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[17] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[18] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[19] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[20] validation_0-error:0.008791 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[21] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.005348 validation_1-f1_err:0.018868
[22] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[23] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[24] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[25] validation_0-error:0.006593 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[26] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[27] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003584 validation_1-f1_err:0.018868
[28] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003584 validation_1-f1_err:0.018868
[29] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.003571 validation_1-f1_err:0.018868
[30] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.001789 validation_1-f1_err:0.018868
[31] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.001789 validation_1-f1_err:0.018868
[32] validation_0-error:0.004396 validation_1-error:0.052632 validation_0-f1_err:0.001789 validation_1-f1_err:0.018868
Stopping. Best iteration:
[7] validation_0-error:0.017582 validation_1-error:0.04386 validation_0-f1_err:0.010638 validation_1-f1_err:0.018868
XGBClassifier(base_score=0.6131868131868132, booster='gbtree',
colsample_bylevel=1, colsample_bytree=0.8, gamma=0,
learning_rate=0.1, max_delta_step=0, max_depth=6,
min_child_weight=1, missing=None, n_estimators=100, n_jobs=8,
niterations=1000, nthread=None, objective='binary:logistic',
random_state=0, reg_alpha=0, reg_lambda=1,
scale_pos_weight=0.6308243727598566, seed=None, silent=False,
subsample=0.8)
你得到 4 个评估矩阵，因为 xgboost 以某种方式向你的 eval_set 添加了另一个评估指标。就个人而言，我使用的是核心 xgboost 而不是 scikit warp up。因此，如果您想了解更多信息，请阅读文档。
对于early_stopping，您必须将n_estimators=1000（或您想要的迭代次数）设置为xgb.XGBClassifier
并在 clf.fit 中设置 early_stopping_rounds=50（或您想要的任何值）。
Here's the documentation。
快速了解提前停止的作用
提前停止决定何时需要停止增强算法以避免过度拟合。它通过评估您在 eval_set 中定义的 tuple (X_test, y_test) 来实现。如果评估错误在 50 次迭代中没有减少，early_stopping 将停止提升。