XGBRegressor:如何修复爆炸 train/val 损失(和无效 random_state)?

XGBRegressor: how to fix exploding train/val loss (and effectless random_state)?

我使用 XGBRegressor 来拟合一个小数据集,(data_size, feature_size) = (156, 328)。虽然给出了random_state,但每次执行程序都无法重现train/val历史记录,有时训练过程很好,有时会出现train/val爆炸问题。为什么 random_state 没有效果?我该如何解决爆炸损失问题?

代码:

from sklearn.model_selection import train_test_split
from xgboost import XGBRegressor
import pandas as pd

SEED = 123
dfs = pd.read_csv('./dir')
dfs_train, dfs_cv = train_test_split(dfs, train_size=0.8, shuffle=False, random_state=SEED)
df_train_x = dfs_train.drop(columns='Y')
df_train_y = dfs_train['Y']
df_cv_x = dfs_cv.drop(columns='Y')
df_cv_y = dfs_cv['Y']


params = {"booster":"gblinear",
          "eval_metric": "rmse",
          "predictor": "cpu_predictor",
          "max_depth": 16,
          "n_estimators":100,
          'random_state':SEED
         }
model = XGBRegressor(**params)
model.fit(df_train_x.values, df_train_y.values,
        eval_set=[(df_train_x.values, df_train_y.values), (df_cv_x.values, df_cv_y.values)],
        eval_metric='rmse',
        verbose=True)

输出 1(爆炸):

[0] validation_0-rmse:1.75475   validation_1-rmse:1.88660
[1] validation_0-rmse:1.25838   validation_1-rmse:1.67099
[2] validation_0-rmse:1.09559   validation_1-rmse:1.52534
[3] validation_0-rmse:1.13592   validation_1-rmse:1.36564
[4] validation_0-rmse:1.17923   validation_1-rmse:1.18143
[5] validation_0-rmse:1.02157   validation_1-rmse:1.34878
[6] validation_0-rmse:0.83439   validation_1-rmse:1.26116
[7] validation_0-rmse:0.75650   validation_1-rmse:1.32562
[8] validation_0-rmse:0.69412   validation_1-rmse:1.26147
[9] validation_0-rmse:0.65568   validation_1-rmse:1.11168
[10]    validation_0-rmse:0.62501   validation_1-rmse:1.13932
[11]    validation_0-rmse:0.61957   validation_1-rmse:1.17217
[12]    validation_0-rmse:0.58313   validation_1-rmse:1.17873
[13]    validation_0-rmse:0.69826   validation_1-rmse:1.28131
[14]    validation_0-rmse:0.65318   validation_1-rmse:1.23954
[15]    validation_0-rmse:0.69506   validation_1-rmse:1.17325
[16]    validation_0-rmse:0.90857   validation_1-rmse:1.16924
[17]    validation_0-rmse:1.29021   validation_1-rmse:1.23918
[18]    validation_0-rmse:0.86403   validation_1-rmse:1.10940
[19]    validation_0-rmse:0.74296   validation_1-rmse:1.09483
[20]    validation_0-rmse:0.66514   validation_1-rmse:1.03155
[21]    validation_0-rmse:0.60940   validation_1-rmse:0.97993
[22]    validation_0-rmse:0.57345   validation_1-rmse:0.91434
[23]    validation_0-rmse:0.56455   validation_1-rmse:0.95662
[24]    validation_0-rmse:0.51317   validation_1-rmse:0.91908
[25]    validation_0-rmse:0.61795   validation_1-rmse:1.19921
[26]    validation_0-rmse:0.52034   validation_1-rmse:0.96785
[27]    validation_0-rmse:0.79248   validation_1-rmse:1.32662
[28]    validation_0-rmse:0.61955   validation_1-rmse:1.02642
[29]    validation_0-rmse:0.59526   validation_1-rmse:1.12646
[30]    validation_0-rmse:0.78931   validation_1-rmse:1.28633
[31]    validation_0-rmse:0.50458   validation_1-rmse:1.08621
[32]    validation_0-rmse:0.83105   validation_1-rmse:1.56490
[33]    validation_0-rmse:0.62568   validation_1-rmse:1.38425
[34]    validation_0-rmse:0.59277   validation_1-rmse:1.32925
[35]    validation_0-rmse:0.54544   validation_1-rmse:1.30204
[36]    validation_0-rmse:0.54612   validation_1-rmse:1.34128
[37]    validation_0-rmse:0.54343   validation_1-rmse:1.36388
[38]    validation_0-rmse:2.05047   validation_1-rmse:2.63729
[39]    validation_0-rmse:7.35043   validation_1-rmse:7.61231
[40]    validation_0-rmse:6.88989   validation_1-rmse:5.74990
[41]    validation_0-rmse:6.68002   validation_1-rmse:6.98875
[42]    validation_0-rmse:8.64272   validation_1-rmse:6.07278
[43]    validation_0-rmse:5.42061   validation_1-rmse:4.87993
[44]    validation_0-rmse:6.02975   validation_1-rmse:6.28529
[45]    validation_0-rmse:5.53219   validation_1-rmse:6.61440
[46]    validation_0-rmse:21.73743  validation_1-rmse:12.64479
[47]    validation_0-rmse:14.01517  validation_1-rmse:15.05459
[48]    validation_0-rmse:9.78612   validation_1-rmse:12.35174
[49]    validation_0-rmse:8.14741   validation_1-rmse:10.34468
[50]    validation_0-rmse:7.37258   validation_1-rmse:9.14025
[51]    validation_0-rmse:13.28054  validation_1-rmse:15.57369
[52]    validation_0-rmse:9.72434   validation_1-rmse:8.82560
[53]    validation_0-rmse:7.43478   validation_1-rmse:8.69813
[54]    validation_0-rmse:6.99072   validation_1-rmse:7.90911
[55]    validation_0-rmse:6.33418   validation_1-rmse:7.16309
[56]    validation_0-rmse:5.98817   validation_1-rmse:6.86138
[57]    validation_0-rmse:6.63810   validation_1-rmse:7.32003
[58]    validation_0-rmse:12.34689  validation_1-rmse:17.12449
[59]    validation_0-rmse:11.46232  validation_1-rmse:11.11735
[60]    validation_0-rmse:8.22308   validation_1-rmse:8.42130
[61]    validation_0-rmse:8.03585   validation_1-rmse:9.78268
[62]    validation_0-rmse:6.08736   validation_1-rmse:9.08017
[63]    validation_0-rmse:5.65990   validation_1-rmse:9.01591
[64]    validation_0-rmse:4.94540   validation_1-rmse:8.60943
[65]    validation_0-rmse:12.16186  validation_1-rmse:9.97841
[66]    validation_0-rmse:24.36063  validation_1-rmse:30.90603
[67]    validation_0-rmse:23.63998  validation_1-rmse:15.92554
[68]    validation_0-rmse:38.54043  validation_1-rmse:49.15125
[69]    validation_0-rmse:26.96050  validation_1-rmse:35.93348
[70]    validation_0-rmse:36.68499  validation_1-rmse:35.61835
[71]    validation_0-rmse:44.18962  validation_1-rmse:41.25709
[72]    validation_0-rmse:35.57274  validation_1-rmse:36.54894
[73]    validation_0-rmse:32.26445  validation_1-rmse:37.02519
[74]    validation_0-rmse:38.02793  validation_1-rmse:60.88339
[75]    validation_0-rmse:29.93598  validation_1-rmse:46.07689
[76]    validation_0-rmse:26.86872  validation_1-rmse:41.39200
[77]    validation_0-rmse:24.87459  validation_1-rmse:41.77614
[78]    validation_0-rmse:29.63828  validation_1-rmse:27.51796
[79]    validation_0-rmse:23.43373  validation_1-rmse:36.54044
[80]    validation_0-rmse:21.80307  validation_1-rmse:38.42451
[81]    validation_0-rmse:45.01890  validation_1-rmse:63.13959
[82]    validation_0-rmse:32.98600  validation_1-rmse:48.51588
[83]    validation_0-rmse:1154.83826    validation_1-rmse:1046.83862
[84]    validation_0-rmse:596.76422 validation_1-rmse:899.20294
[85]    validation_0-rmse:8772.32227    validation_1-rmse:14788.31152
[86]    validation_0-rmse:15234.09082   validation_1-rmse:14237.62500
[87]    validation_0-rmse:12527.86426   validation_1-rmse:13914.09277
[88]    validation_0-rmse:11000.84277   validation_1-rmse:13445.76074
[89]    validation_0-rmse:15696.28613   validation_1-rmse:13946.85840
[90]    validation_0-rmse:85210.62500   validation_1-rmse:127271.79688
[91]    validation_0-rmse:116500.62500  validation_1-rmse:215355.65625
[92]    validation_0-rmse:149855.62500  validation_1-rmse:147734.62500
[93]    validation_0-rmse:151028.76562  validation_1-rmse:97522.35938
[94]    validation_0-rmse:286164.06250  validation_1-rmse:359728.84375
[95]    validation_0-rmse:149474.23438  validation_1-rmse:182052.50000
[96]    validation_0-rmse:156148.78125  validation_1-rmse:217708.90625
[97]    validation_0-rmse:114551.62500  validation_1-rmse:151682.79688
[98]    validation_0-rmse:104612.85156  validation_1-rmse:170244.31250
[99]    validation_0-rmse:256178.57812  validation_1-rmse:246638.64062

输出 2(精细):

[0] validation_0-rmse:2.73642   validation_1-rmse:2.73807
[1] validation_0-rmse:0.49221   validation_1-rmse:0.80462
[2] validation_0-rmse:0.31022   validation_1-rmse:0.73898
[3] validation_0-rmse:0.26974   validation_1-rmse:0.76231
[4] validation_0-rmse:0.22617   validation_1-rmse:0.61529
[5] validation_0-rmse:0.20344   validation_1-rmse:0.66840
[6] validation_0-rmse:0.18369   validation_1-rmse:0.62763
[7] validation_0-rmse:0.17476   validation_1-rmse:0.64966
[8] validation_0-rmse:0.16620   validation_1-rmse:0.60988
[9] validation_0-rmse:0.16017   validation_1-rmse:0.62756
[10]    validation_0-rmse:0.15479   validation_1-rmse:0.61354
[11]    validation_0-rmse:0.15247   validation_1-rmse:0.63041
[12]    validation_0-rmse:0.14641   validation_1-rmse:0.58863
[13]    validation_0-rmse:0.14544   validation_1-rmse:0.55724
[14]    validation_0-rmse:0.16165   validation_1-rmse:0.54285
[15]    validation_0-rmse:0.14305   validation_1-rmse:0.59282
[16]    validation_0-rmse:0.13728   validation_1-rmse:0.57130
[17]    validation_0-rmse:0.13325   validation_1-rmse:0.56199
[18]    validation_0-rmse:0.12974   validation_1-rmse:0.53802
[19]    validation_0-rmse:0.12596   validation_1-rmse:0.54721
[20]    validation_0-rmse:0.12342   validation_1-rmse:0.54109
[21]    validation_0-rmse:0.12143   validation_1-rmse:0.53365
[22]    validation_0-rmse:0.11954   validation_1-rmse:0.53702
[23]    validation_0-rmse:0.11721   validation_1-rmse:0.52632
[24]    validation_0-rmse:0.11521   validation_1-rmse:0.52671
[25]    validation_0-rmse:0.11325   validation_1-rmse:0.51527
[26]    validation_0-rmse:0.11148   validation_1-rmse:0.51392
[27]    validation_0-rmse:0.10978   validation_1-rmse:0.49357
[28]    validation_0-rmse:0.10803   validation_1-rmse:0.50030
[29]    validation_0-rmse:0.10657   validation_1-rmse:0.49821
[30]    validation_0-rmse:0.10624   validation_1-rmse:0.47754
[31]    validation_0-rmse:0.10450   validation_1-rmse:0.48614
[32]    validation_0-rmse:0.10336   validation_1-rmse:0.47555
[33]    validation_0-rmse:0.10213   validation_1-rmse:0.47663
[34]    validation_0-rmse:0.10139   validation_1-rmse:0.47462
[35]    validation_0-rmse:0.09979   validation_1-rmse:0.46085
[36]    validation_0-rmse:0.09875   validation_1-rmse:0.46658
[37]    validation_0-rmse:0.09780   validation_1-rmse:0.46026
[38]    validation_0-rmse:0.09702   validation_1-rmse:0.45724
[39]    validation_0-rmse:0.09638   validation_1-rmse:0.46206
[40]    validation_0-rmse:0.09570   validation_1-rmse:0.46017
[41]    validation_0-rmse:0.09500   validation_1-rmse:0.45447
[42]    validation_0-rmse:0.09431   validation_1-rmse:0.45097
[43]    validation_0-rmse:0.09371   validation_1-rmse:0.45112
[44]    validation_0-rmse:0.09322   validation_1-rmse:0.44389
[45]    validation_0-rmse:0.09271   validation_1-rmse:0.45073
[46]    validation_0-rmse:0.09199   validation_1-rmse:0.44402
[47]    validation_0-rmse:0.09145   validation_1-rmse:0.44305
[48]    validation_0-rmse:0.09091   validation_1-rmse:0.43982
[49]    validation_0-rmse:0.09028   validation_1-rmse:0.43441
[50]    validation_0-rmse:0.09004   validation_1-rmse:0.44175
[51]    validation_0-rmse:0.08931   validation_1-rmse:0.43299
[52]    validation_0-rmse:0.09034   validation_1-rmse:0.41695
[53]    validation_0-rmse:0.08860   validation_1-rmse:0.41444
[54]    validation_0-rmse:0.08798   validation_1-rmse:0.40965
[55]    validation_0-rmse:0.08734   validation_1-rmse:0.41013
[56]    validation_0-rmse:0.08744   validation_1-rmse:0.39615
[57]    validation_0-rmse:0.08636   validation_1-rmse:0.40437
[58]    validation_0-rmse:0.08597   validation_1-rmse:0.40617
[59]    validation_0-rmse:0.08559   validation_1-rmse:0.40638
[60]    validation_0-rmse:0.08518   validation_1-rmse:0.41139
[61]    validation_0-rmse:0.08472   validation_1-rmse:0.40855
[62]    validation_0-rmse:0.08427   validation_1-rmse:0.40601
[63]    validation_0-rmse:0.08386   validation_1-rmse:0.40446
[64]    validation_0-rmse:0.08357   validation_1-rmse:0.40676
[65]    validation_0-rmse:0.08347   validation_1-rmse:0.39509
[66]    validation_0-rmse:0.08295   validation_1-rmse:0.40182
[67]    validation_0-rmse:0.08269   validation_1-rmse:0.40343
[68]    validation_0-rmse:0.08294   validation_1-rmse:0.39187
[69]    validation_0-rmse:0.08231   validation_1-rmse:0.39857
[70]    validation_0-rmse:0.08200   validation_1-rmse:0.39805
[71]    validation_0-rmse:0.08178   validation_1-rmse:0.39975
[72]    validation_0-rmse:0.08200   validation_1-rmse:0.40522
[73]    validation_0-rmse:0.08104   validation_1-rmse:0.40048
[74]    validation_0-rmse:0.08073   validation_1-rmse:0.39871
[75]    validation_0-rmse:0.08041   validation_1-rmse:0.39395
[76]    validation_0-rmse:0.08022   validation_1-rmse:0.39725
[77]    validation_0-rmse:0.07989   validation_1-rmse:0.39610
[78]    validation_0-rmse:0.07964   validation_1-rmse:0.39375
[79]    validation_0-rmse:0.07942   validation_1-rmse:0.38979
[80]    validation_0-rmse:0.07920   validation_1-rmse:0.39015
[81]    validation_0-rmse:0.07914   validation_1-rmse:0.38749
[82]    validation_0-rmse:0.07890   validation_1-rmse:0.38585
[83]    validation_0-rmse:0.07868   validation_1-rmse:0.38665
[84]    validation_0-rmse:0.07842   validation_1-rmse:0.38147
[85]    validation_0-rmse:0.07819   validation_1-rmse:0.38246
[86]    validation_0-rmse:0.07805   validation_1-rmse:0.38351
[87]    validation_0-rmse:0.07796   validation_1-rmse:0.37884
[88]    validation_0-rmse:0.07770   validation_1-rmse:0.38242
[89]    validation_0-rmse:0.07750   validation_1-rmse:0.37763
[90]    validation_0-rmse:0.07724   validation_1-rmse:0.37871
[91]    validation_0-rmse:0.07702   validation_1-rmse:0.37974
[92]    validation_0-rmse:0.07679   validation_1-rmse:0.38147
[93]    validation_0-rmse:0.07664   validation_1-rmse:0.37735
[94]    validation_0-rmse:0.07644   validation_1-rmse:0.37873
[95]    validation_0-rmse:0.07632   validation_1-rmse:0.37661
[96]    validation_0-rmse:0.07610   validation_1-rmse:0.37877
[97]    validation_0-rmse:0.07587   validation_1-rmse:0.37659
[98]    validation_0-rmse:0.07572   validation_1-rmse:0.37648
[99]    validation_0-rmse:0.07556   validation_1-rmse:0.37356

已更新:

通过使用 public 波士顿住房数据集并设置 nthread=1,训练过程变得可重现,而不会出现爆炸性问题。看来问题出在我的数据集中。代码和输出如下:

代码:

from sklearn.datasets import load_boston
import sklearn 
from xgboost import XGBRegressor
import pandas as pd
import numpy as np

SEED = 123

X, y = load_boston(return_X_y=True)
np.random.seed(SEED)
indices = np.random.permutation(X.shape[0])
training_idx, test_idx = indices[:80], indices[80:]
train_X, test_X = X[training_idx,:], X[test_idx,:]
train_y, test_y = y[training_idx], y[test_idx]


params = {"booster":"gblinear",
          "eval_metric": "rmse",
          "predictor": "cpu_predictor",
          "max_depth": 16,
          "n_estimators":100,
          'random_state':SEED,
          'nthread':1,
          'early_stopping_rounds':5
         }

model = XGBRegressor(**params)
model.get_xgb_params()

model.fit(train_X, train_y,
        eval_set=[(train_X, train_y), (test_X, test_y)],
        eval_metric='rmse',
        verbose=True)

输出:

{'objective': 'reg:squarederror',
 'base_score': None,
 'booster': 'gblinear',
 'colsample_bylevel': None,
 'colsample_bynode': None,
 'colsample_bytree': None,
 'gamma': None,
 'gpu_id': None,
 'interaction_constraints': None,
 'learning_rate': None,
 'max_delta_step': None,
 'max_depth': 16,
 'min_child_weight': None,
 'monotone_constraints': None,
 'n_jobs': None,
 'num_parallel_tree': None,
 'random_state': 123,
 'reg_alpha': None,
 'reg_lambda': None,
 'scale_pos_weight': None,
 'subsample': None,
 'tree_method': None,
 'validate_parameters': None,
 'verbosity': None,
 'eval_metric': 'rmse',
 'predictor': 'cpu_predictor',
 'nthread': 1,
 'early_stopping_rounds': 5}
Parameters: { early_stopping_rounds, max_depth, predictor } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.


[0] validation_0-rmse:8.38695   validation_1-rmse:8.88360
[1] validation_0-rmse:7.56356   validation_1-rmse:8.06591
[2] validation_0-rmse:7.24844   validation_1-rmse:7.71700
[3] validation_0-rmse:7.03799   validation_1-rmse:7.46547
[4] validation_0-rmse:6.86494   validation_1-rmse:7.25173
[5] validation_0-rmse:6.71517   validation_1-rmse:7.06397
[6] validation_0-rmse:6.58385   validation_1-rmse:6.89819
[7] validation_0-rmse:6.46814   validation_1-rmse:6.75184
[8] validation_0-rmse:6.36585   validation_1-rmse:6.62274
[9] validation_0-rmse:6.27512   validation_1-rmse:6.50893
[10]    validation_0-rmse:6.19437   validation_1-rmse:6.40863
[11]    validation_0-rmse:6.12223   validation_1-rmse:6.32025
[12]    validation_0-rmse:6.05754   validation_1-rmse:6.24240
[13]    validation_0-rmse:5.99930   validation_1-rmse:6.17386
[14]    validation_0-rmse:5.94666   validation_1-rmse:6.11355
[15]    validation_0-rmse:5.89892   validation_1-rmse:6.06053
[16]    validation_0-rmse:5.85546   validation_1-rmse:6.01398
[17]    validation_0-rmse:5.81576   validation_1-rmse:5.97318
[18]    validation_0-rmse:5.77938   validation_1-rmse:5.93750
[19]    validation_0-rmse:5.74595   validation_1-rmse:5.90638
[20]    validation_0-rmse:5.71514   validation_1-rmse:5.87933
[21]    validation_0-rmse:5.68669   validation_1-rmse:5.85592
[22]    validation_0-rmse:5.66035   validation_1-rmse:5.83575
[23]    validation_0-rmse:5.63591   validation_1-rmse:5.81850
[24]    validation_0-rmse:5.61321   validation_1-rmse:5.80385
[25]    validation_0-rmse:5.59208   validation_1-rmse:5.79153
[26]    validation_0-rmse:5.57239   validation_1-rmse:5.78130
[27]    validation_0-rmse:5.55401   validation_1-rmse:5.77294
[28]    validation_0-rmse:5.53685   validation_1-rmse:5.76626
[29]    validation_0-rmse:5.52081   validation_1-rmse:5.76107
[30]    validation_0-rmse:5.50579   validation_1-rmse:5.75723
[31]    validation_0-rmse:5.49174   validation_1-rmse:5.75458
[32]    validation_0-rmse:5.47856   validation_1-rmse:5.75300
[33]    validation_0-rmse:5.46621   validation_1-rmse:5.75237
[34]    validation_0-rmse:5.45463   validation_1-rmse:5.75258
[35]    validation_0-rmse:5.44376   validation_1-rmse:5.75354
[36]    validation_0-rmse:5.43355   validation_1-rmse:5.75516
[37]    validation_0-rmse:5.42396   validation_1-rmse:5.75736
[38]    validation_0-rmse:5.41496   validation_1-rmse:5.76008
[39]    validation_0-rmse:5.40649   validation_1-rmse:5.76324
[40]    validation_0-rmse:5.39853   validation_1-rmse:5.76679
[41]    validation_0-rmse:5.39104   validation_1-rmse:5.77068
[42]    validation_0-rmse:5.38399   validation_1-rmse:5.77484
[43]    validation_0-rmse:5.37735   validation_1-rmse:5.77926
[44]    validation_0-rmse:5.37111   validation_1-rmse:5.78387
[45]    validation_0-rmse:5.36522   validation_1-rmse:5.78865
[46]    validation_0-rmse:5.35967   validation_1-rmse:5.79357
[47]    validation_0-rmse:5.35443   validation_1-rmse:5.79859
[48]    validation_0-rmse:5.34949   validation_1-rmse:5.80369
[49]    validation_0-rmse:5.34483   validation_1-rmse:5.80885
[50]    validation_0-rmse:5.34042   validation_1-rmse:5.81403
[51]    validation_0-rmse:5.33626   validation_1-rmse:5.81924
[52]    validation_0-rmse:5.33231   validation_1-rmse:5.82444
[53]    validation_0-rmse:5.32859   validation_1-rmse:5.82962
[54]    validation_0-rmse:5.32505   validation_1-rmse:5.83477
[55]    validation_0-rmse:5.32170   validation_1-rmse:5.83988
[56]    validation_0-rmse:5.31853   validation_1-rmse:5.84493
[57]    validation_0-rmse:5.31551   validation_1-rmse:5.84992
[58]    validation_0-rmse:5.31265   validation_1-rmse:5.85483
[59]    validation_0-rmse:5.30992   validation_1-rmse:5.85966
[60]    validation_0-rmse:5.30732   validation_1-rmse:5.86441
[61]    validation_0-rmse:5.30485   validation_1-rmse:5.86906
[62]    validation_0-rmse:5.30249   validation_1-rmse:5.87362
[63]    validation_0-rmse:5.30024   validation_1-rmse:5.87808
[64]    validation_0-rmse:5.29808   validation_1-rmse:5.88244
[65]    validation_0-rmse:5.29602   validation_1-rmse:5.88668
[66]    validation_0-rmse:5.29404   validation_1-rmse:5.89082
[67]    validation_0-rmse:5.29215   validation_1-rmse:5.89485
[68]    validation_0-rmse:5.29033   validation_1-rmse:5.89877
[69]    validation_0-rmse:5.28858   validation_1-rmse:5.90257
[70]    validation_0-rmse:5.28690   validation_1-rmse:5.90626
[71]    validation_0-rmse:5.28527   validation_1-rmse:5.90984
[72]    validation_0-rmse:5.28371   validation_1-rmse:5.91331
[73]    validation_0-rmse:5.28219   validation_1-rmse:5.91666
[74]    validation_0-rmse:5.28073   validation_1-rmse:5.91990
[75]    validation_0-rmse:5.27931   validation_1-rmse:5.92303
[76]    validation_0-rmse:5.27794   validation_1-rmse:5.92605
[77]    validation_0-rmse:5.27661   validation_1-rmse:5.92896
[78]    validation_0-rmse:5.27531   validation_1-rmse:5.93176
[79]    validation_0-rmse:5.27405   validation_1-rmse:5.93445
[80]    validation_0-rmse:5.27282   validation_1-rmse:5.93704
[81]    validation_0-rmse:5.27163   validation_1-rmse:5.93953
[82]    validation_0-rmse:5.27046   validation_1-rmse:5.94192
[83]    validation_0-rmse:5.26932   validation_1-rmse:5.94420
[84]    validation_0-rmse:5.26820   validation_1-rmse:5.94639
[85]    validation_0-rmse:5.26711   validation_1-rmse:5.94848
[86]    validation_0-rmse:5.26604   validation_1-rmse:5.95048
[87]    validation_0-rmse:5.26499   validation_1-rmse:5.95238
[88]    validation_0-rmse:5.26396   validation_1-rmse:5.95420
[89]    validation_0-rmse:5.26294   validation_1-rmse:5.95592
[90]    validation_0-rmse:5.26195   validation_1-rmse:5.95756
[91]    validation_0-rmse:5.26097   validation_1-rmse:5.95912
[92]    validation_0-rmse:5.26000   validation_1-rmse:5.96059
[93]    validation_0-rmse:5.25905   validation_1-rmse:5.96198
[94]    validation_0-rmse:5.25811   validation_1-rmse:5.96329
[95]    validation_0-rmse:5.25718   validation_1-rmse:5.96453
[96]    validation_0-rmse:5.25627   validation_1-rmse:5.96569
[97]    validation_0-rmse:5.25537   validation_1-rmse:5.96678
[98]    validation_0-rmse:5.25447   validation_1-rmse:5.96779
[99]    validation_0-rmse:5.25359   validation_1-rmse:5.96874

奇怪。以下是我要尝试的顺序:

  1. 在开始训练之前,在 XGBRegressor 模型上使用 get_params/get_xgb_params() 以确保它确实使用了您传入的 random_state 参数。同上,查看详细日志以确保肯定训练用过它。
  2. 查看目标变量y。它的分布是不是很怪异,不连续?请给我们看一个图或直方图?或至少一些汇总统计数据(最小值、最大值、平均值、中值、标准差、第一和第三四分位数)?未分层拆分会影响您的训练吗? (显示拆分前后的描述性统计数据,也在 eval 集上,这三个集应该没有太大差异)。尝试对 log(y)、sqrt(y)、exp(y) 或类似的模型建模是否更容易?您可以调试哪些行导致 CV 错误吗?
    • 可能eval_metric='rmse'和默认objectivereg:squarederror非常不适合你的目标变量y。我们应该能够从 y 的图中分辨出来,但请尝试其他 [eval_metrics 和 objectives] (https://xgboost.readthedocs.io/en/latest/parameter.html#learning-task-parameters),例如 eval_metric=='logloss''mae''rmsle',等等。请参阅该文档以获取 objectiveeval_metric.
    • 的完整列表
  3. 此外,为了完全确定性,将 nthread 参数设置为 1(单核)。默认值为 nthread==-1(使用所有核心)。然后重新运行运行 1 和 2 并更新问题中的结果。
  4. 失败了,你能做一个可重现的例子吗(MCVE)?请告诉我们您的数据集在哪里公开可用(来源 URL,而不是云 link),或者使用任何公开可用的数据集制作一个可重现的示例?