情绪分析 - 交叉验证无效分数

Question

我正在使用 NLTK 和 SKlearn 测试情感分析模型。

Movie_reviews 数据有 "pos" 和 "neg" 标签。为了训练分类器，我使用 "featuresets"。我正在对训练数据和测试数据的准确性使用交叉验证。然而，交叉验证总是比准确性高得多。在下面的逻辑回归算法示例中，CV = 97（平均值），准确度 = 70。我也用其他算法进行了测试，交叉验证仍然非常高。

我很确定我的交叉验证代码不正确。

import nltk
import random 
from nltk.corpus import movie_reviews  
from sklearn import cross_validation
from nltk.classify.scikitlearn import SklearnClassifier
from sklearn.linear_model import LogisticRegression, SGDClassifier

documents = [(list(movie_reviews.words(fileid)), category)
             for category in movie_reviews.categories()
             for fileid in movie_reviews.fileids(category)]
random.shuffle(documents)

all_words = []
for w in movie_reviews.words():
    all_words.append(w.lower())

all_words = nltk.FreqDist(all_words)
word_features = list(all_words.keys())[:3000]

def find_features(document):
    words = set(document)
    features = {}
    for w in word_features:
        features[w] = (w in words)
    return features

featuresets = [(find_features(rev), category) for (rev, category) in documents]        
training_set = featuresets[:1500]
testing_set =  featuresets[1500:]

cv = cross_validation.KFold(len(training_set), n_folds=10, shuffle=True, random_state=None)
LogisticRegression_classifier = SklearnClassifier(LogisticRegression())    
for traincv, testcv in cv:
    classifier = LogisticRegression_classifier.train(training_set[traincv[0]:traincv[len(traincv)-1]])
    print ('CV_accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv[0]:testcv[len(testcv)-1]]))

print("LogisticRegression_classifier accuracy percent:", (nltk.classify.accuracy(LogisticRegression_classifier, testing_set))*100)

Answer 1

您正在使用 training_set[traincv[0]:traincv[len(traincv)-1]] 这意味着从 traincv[0] 到 traincv[ len(traincv)-1]

在您的情况下，traincv[0] 和 testcv[0] 将始终接近 0，traincv[len(traincv)-1] 和 testcv[len(testcv)-1] 将始终接近 1499。所以您在进行 N 倍验证时使用几乎相同的数据进行训练和测试。

在这里，你实际上需要使用traincv和testcv中的子集索引。

import numpy as np
training_set = np.array(training_set)
for traincv, testcv in cv:
    classifier = LogisticRegression_classifier.train(training_set[traincv])
    print ('CV_accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv]

情绪分析 - 交叉验证无效分数

Sentiment Analysis - Cross Validation not valid score

python

scikit-learn

nltk

cross-validation