交叉验证分类错误

Question

我正在使用以下代码来获取分类结果：

 folds = 5 #number of folds for the cv

        #Logistic Regression--
        clf = linear_model.LogisticRegression(penalty='l1')
        kf = KFold

(len(clas), n_folds=folds)
    fold = 1
    cms = np.array([[0,0],[0,0]])
    accs = []
    aucs=[]
    for train_index, test_index in kf:
        X_train, X_test = docs[train_index], docs[test_index]
        y_train, y_test = clas2[train_index], clas2[test_index]
        clf.fit(X_train, y_train)
        prediction = clf.predict(X_test)
        acc = accuracy_score(prediction, y_test)
        cm = confusion_matrix(y_test,prediction)
        pred_probas = clf.predict_proba(X_test)[:,1]
        fpr, tpr, thresholds = metrics.roc_curve(y_test, pred_probas)
        print('Test Accuracy for fold {}: {}\n{}'.format(fold,round((acc*100),2),cm))
        roc_auc = auc(fpr,tpr)
        print('AUC for fold {} : {}'.format(fold,round((roc_auc*100),2)))
        fold +=1
        cms += cm
        accs.append(acc)
        aucs.append(roc_auc)
    print('CV test accuracy: {}\n{}'.format(round((np.mean(accs)*100),2),cms))
    print('\nCV AUC: {}'.format(round(np.mean(aucs)*100),2))
    print('\nCV accuracy: %.3f +/- %.3f' % (round((np.mean(accs)*100),2),round((np.std(accs)*100),2)))
    print('\nCV ROC AUC: %.3f +/- %.3f' % (round((np.mean(aucs)*100),2),round((np.std(aucs)*100),2)))
    print('\nPeak accuracy: '+str(round((np.amax(accs)*100),2)))
    print('\nPeak ROC AUC: '+str(round((np.amax(aucs)*100),2)))

我不确定我是否正在做某事但我有 2 类是= 406 No= 139，代码给出了以下结果

Test Accuracy for fold 1: 87.16
[[94  9]
 [ 5  1]]
AUC for fold 1 : 66.1
Test Accuracy for fold 2: 92.66
[[100   6]
 [  2   1]]
AUC for fold 2 : 62.42
Test Accuracy for fold 3: 90.83
[[99  7]
 [ 3  0]]
AUC for fold 3 : 43.08
Test Accuracy for fold 4: 88.07
[[83  8]
 [ 5 13]]
AUC for fold 4 : 85.5
Test Accuracy for fold 5: 53.21
[[ 0  0]
 [51 58]]
AUC for fold 5 : nan
CV test accuracy: 82.39
[[376  30]
 [ 66  73]]

CV AUC: nan

CV accuracy: 82.390 +/- 14.720

CV ROC AUC: nan +/- nan

Peak accuracy: 92.66

Peak ROC AUC: nan
C:\Users\kkothari\AppData\Local\Continuum\Anaconda3\lib\site-packages\sklearn\metrics\ranking.py:530: UndefinedMetricWarning: No negative samples in y_true, false positive value should be meaningless
  UndefinedMetricWarning)
C:\Users\kkothari\AppData\Local\Continuum\Anaconda3\lib\site-packages\sklearn\metrics\ranking.py:95: RuntimeWarning: invalid value encountered in less
  if np.any(dx < 0):

最初我只有 17 没有文档，但它工作正常..有人可以指出一些错误或解释发生了什么吗？

Answer 1

基本上你有一个非常小的 class（大约 20-30 个样本？）并且在其中一个拆分中你没有得到任何，因此导致错误。您可以改用 StratifiedKFold，它保证在每个拆分中您都有来自每个 class.

的恒定数量的样本

交叉验证分类错误

Cross Validation classification error

python

machine-learning

scikit-learn

cross-validation

text-classification