Word2Vec - 具有高交叉验证分数的模型对测试数据的表现非常糟糕
Word2Vec - Model with high cross validation score performs incredibly bad for test data
在对推特数据进行情感分析时，我遇到了一个我无法解决的问题。我想训练一个 RandomForest 分类器来检测仇恨言论。因此，我使用了一个带标签的数据集，其中的推文被标记为 1 表示仇恨言论，0 表示正常推文。对于矢量化，我使用的是 Word2Vec。我首先执行超参数化来为分类器找到好的参数。
在超参数化过程中，我使用了重复的分层 KFold 交叉验证（评分 = 准确性）
这里的平均准确度约为 99.6%。然而，一旦我将该模型应用于测试数据集并绘制混淆矩阵，准确率仅高于 50%，这对于二元分类器来说当然很糟糕。
我成功地将完全相同的方法用于 Bag of Words，并且在这里完全没有问题。
有人可以快速查看我的代码吗？那会很有帮助。我就是找不到问题所在。非常感谢！
（我也将代码上传到 google collab 以防你更容易：https://colab.research.google.com/drive/15BzElijL3vwa_6DnLicxRvcs4SPDZbpe?usp=sharing ）
首先我预处理了我的数据：
train_csv = pd.read_csv(r'/content/drive/My Drive/Colab Notebooks/MLDA_project/data2/train.csv')
train = train_csv
#check for missing values (result shows that there are no missing values)
train.isna().sum()
# remove the tweet IDs
train.drop(train.columns[0], axis = "columns", inplace = True)
# create a new column to save the cleansed tweets
train['training_tweet'] = np.nan
# remove special/unknown characters
train.replace('[^a-zA-Z#]', ' ', inplace = True, regex = True)
# generate stopword list and add the twitter handles "user" to the stopword list
stopwords = sw.words('english')
stopwords.append('user')
# convert to lowercase
train = train.applymap(lambda i:i.lower() if type(i) == str else i)
# execute tokenization and lemmatization
lemmatizer = WordNetLemmatizer()
for i in range(len(train.index)):
#tokenize the tweets from the column "tweet"
words = nltk.word_tokenize(train.iloc[i, 1])
#consider words with more than 3 characters
words = [word for word in words if len(word) > 3]
#exclude words in stopword list
words = [lemmatizer.lemmatize(word) for word in words if word not in set(stopwords)]
#Join words again
train.iloc[i, 2] = ' '.join(words)
words = nltk.word_tokenize(train.iloc[i, 2])
train.drop(train.columns[1], axis = "columns", inplace = True)
majority = train[train.label == 0]
minority = train[train.label == 1]
# upsample minority class
minority_upsampled = resample(minority, replace = True, n_samples = len(majority))
# combine majority class with upsampled minority class
train_upsampled = pd.concat([majority, minority_upsampled])
train = train_upsampled
np.random.seed(10)
train = train.sample(frac = 1)
train = train.reset_index(drop = True)
现在 train 在第 0 列中有标签，在第 1 列中有预处理的推文。
接下来我定义了 Word2Vec Vectorizer：
def W2Vvectorize(X_train):
tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)
vector=[]
from tqdm import tqdm
for sent in tqdm(tokenize):
sent_vec=np.zeros(100)
count =0
for word in sent:
if word in w2v_words:
vec = w2vec_model.wv[word]
sent_vec += vec
count += 1
if count != 0:
sent_vec /= count #normalize
vector.append(sent_vec)
return vector
我将数据集拆分为测试集和训练集，并使用上面定义的 W2V 对两个子集进行矢量化：
x = train["training_tweet"]
y = train["label"]
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])
print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) # change 0,4 & 0,6
train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)
现在我进行超参数化：
# define models and parameters
model = RandomForestClassifier()
n_estimators = [10, 100, 1000]
max_features = ['sqrt', 'log2']
# define grid search
grid = dict(n_estimators=n_estimators,max_features=max_features)
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
grid_search = GridSearchCV(estimator=model, param_grid=grid, n_jobs=-1, cv=cv, scoring='accuracy',error_score=0)
grid_result = grid_search.fit(train_tf_w2v, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
print("%f (%f) with: %r" % (mean, stdev, param))
这导致以下输出：
Best: 0.996628 using {'max_features': 'log2', 'n_estimators': 1000}
0.995261 (0.000990) with: {'max_features': 'sqrt', 'n_estimators': 10}
0.996110 (0.000754) with: {'max_features': 'sqrt', 'n_estimators': 100}
0.996081 (0.000853) with: {'max_features': 'sqrt', 'n_estimators': 1000}
0.995885 (0.000872) with: {'max_features': 'log2', 'n_estimators': 10}
0.996481 (0.000691) with: {'max_features': 'log2', 'n_estimators': 100}
0.996628 (0.000782) with: {'max_features': 'log2', 'n_estimators': 1000}
接下来，我想用模型绘制一个带有测试数据的混淆矩阵：
clf = RandomForestClassifier(max_features = 'log2', n_estimators=1000)
clf.fit(train_tf_w2v, y_train)
name = clf.__class__.__name__
expectation = y_test
test_prediction = clf.predict(test_tf_w2v)
acc = accuracy_score(expectation, test_prediction)
pre = precision_score(expectation, test_prediction)
rec = recall_score(expectation, test_prediction)
f1 = f1_score(expectation, test_prediction)
fig, ax = plt.subplots(1,2, figsize=(14,4))
plt.suptitle(f'{name} \n', fontsize = 18)
plt.subplots_adjust(top = 0.8)
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, ax=ax[0])
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, normalize=True, ax = ax[1])
plt.show()
print(f"for the {name} we receive the following values:")
print("Accuracy: {:.3%}".format(acc))
print('Precision score: {:.3%}'.format(pre))
print('Recall score: {:.3%}'.format(rec))
print('F1 score: {:.3%}'.format(f1))
这输出：
对于 RandomForestClassifier，我们收到以下值：
准确度：57.974%
准确率：99.790%
召回率：15.983%
F1分数：27.552%
哎哟...现在我觉得自己很傻。我发现哪里不对了。
在 train/test-split 之后，我将两个子集独立地发送到 W2Vvectorize() 函数。
train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)
从那里 W2Vvectorize() 函数基于两个独立的子集训练两个独立的 Word2Vec 模型。因此，当我将矢量化测试数据 test_tf_w2v 传递给我训练有素的 RandomForest 分类器时，为了检查测试集的准确性是否也正确，在训练有素的 RandomForest 分类器看来，测试集似乎在不同的语言。两个独立的 word2vec 模型只是以不同的方式向量化。
我按如下方式解决了这个问题：
def W2Vvectorize(X_train):
tokenize=X_train.apply(lambda x: x.split())
vector=[]
for sent in tqdm(tokenize):
sent_vec=np.zeros(100)
count =0
for word in sent:
if word in w2v_words:
vec = w2vec_model.wv[word]
sent_vec += vec
count += 1
if count != 0:
sent_vec /= count #normalize
vector.append(sent_vec)
return vector
并且 Word2Vec 训练与此分开：
x = train["training_tweet"]
y = train["label"]
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])
print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) #
tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)
train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)
因此，Word2Vec 模型训练仅在训练数据上执行。然而，测试数据的矢量化必须使用完全相同的 Word2Vec 模型进行。
只是为了完整性：精度太高的原因是，我平衡了数据集，以便在最终训练集中均匀分布 classes。因此，我用更少的数据对 class 进行了上采样。这当然意味着之后数据集多次包含上采样数据。
如果您像我一样在上采样后拆分测试数据，那么您的训练数据很可能会包含来自上采样数据的元素。我画了一张图来解释：
在对推特数据进行情感分析时，我遇到了一个我无法解决的问题。我想训练一个 RandomForest 分类器来检测仇恨言论。因此，我使用了一个带标签的数据集，其中的推文被标记为 1 表示仇恨言论，0 表示正常推文。对于矢量化，我使用的是 Word2Vec。我首先执行超参数化来为分类器找到好的参数。 在超参数化过程中，我使用了重复的分层 KFold 交叉验证（评分 = 准确性） 这里的平均准确度约为 99.6%。然而，一旦我将该模型应用于测试数据集并绘制混淆矩阵，准确率仅高于 50%，这对于二元分类器来说当然很糟糕。 我成功地将完全相同的方法用于 Bag of Words，并且在这里完全没有问题。 有人可以快速查看我的代码吗？那会很有帮助。我就是找不到问题所在。非常感谢！
（我也将代码上传到 google collab 以防你更容易：https://colab.research.google.com/drive/15BzElijL3vwa_6DnLicxRvcs4SPDZbpe?usp=sharing ）
首先我预处理了我的数据：
train_csv = pd.read_csv(r'/content/drive/My Drive/Colab Notebooks/MLDA_project/data2/train.csv')
train = train_csv
#check for missing values (result shows that there are no missing values)
train.isna().sum()
# remove the tweet IDs
train.drop(train.columns[0], axis = "columns", inplace = True)
# create a new column to save the cleansed tweets
train['training_tweet'] = np.nan
# remove special/unknown characters
train.replace('[^a-zA-Z#]', ' ', inplace = True, regex = True)
# generate stopword list and add the twitter handles "user" to the stopword list
stopwords = sw.words('english')
stopwords.append('user')
# convert to lowercase
train = train.applymap(lambda i:i.lower() if type(i) == str else i)
# execute tokenization and lemmatization
lemmatizer = WordNetLemmatizer()
for i in range(len(train.index)):
#tokenize the tweets from the column "tweet"
words = nltk.word_tokenize(train.iloc[i, 1])
#consider words with more than 3 characters
words = [word for word in words if len(word) > 3]
#exclude words in stopword list
words = [lemmatizer.lemmatize(word) for word in words if word not in set(stopwords)]
#Join words again
train.iloc[i, 2] = ' '.join(words)
words = nltk.word_tokenize(train.iloc[i, 2])
train.drop(train.columns[1], axis = "columns", inplace = True)
majority = train[train.label == 0]
minority = train[train.label == 1]
# upsample minority class
minority_upsampled = resample(minority, replace = True, n_samples = len(majority))
# combine majority class with upsampled minority class
train_upsampled = pd.concat([majority, minority_upsampled])
train = train_upsampled
np.random.seed(10)
train = train.sample(frac = 1)
train = train.reset_index(drop = True)
现在 train 在第 0 列中有标签，在第 1 列中有预处理的推文。
接下来我定义了 Word2Vec Vectorizer：
def W2Vvectorize(X_train):
tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)
vector=[]
from tqdm import tqdm
for sent in tqdm(tokenize):
sent_vec=np.zeros(100)
count =0
for word in sent:
if word in w2v_words:
vec = w2vec_model.wv[word]
sent_vec += vec
count += 1
if count != 0:
sent_vec /= count #normalize
vector.append(sent_vec)
return vector
我将数据集拆分为测试集和训练集，并使用上面定义的 W2V 对两个子集进行矢量化：
x = train["training_tweet"]
y = train["label"]
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])
print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) # change 0,4 & 0,6
train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)
现在我进行超参数化：
# define models and parameters
model = RandomForestClassifier()
n_estimators = [10, 100, 1000]
max_features = ['sqrt', 'log2']
# define grid search
grid = dict(n_estimators=n_estimators,max_features=max_features)
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
grid_search = GridSearchCV(estimator=model, param_grid=grid, n_jobs=-1, cv=cv, scoring='accuracy',error_score=0)
grid_result = grid_search.fit(train_tf_w2v, y_train)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
print("%f (%f) with: %r" % (mean, stdev, param))
这导致以下输出：
Best: 0.996628 using {'max_features': 'log2', 'n_estimators': 1000}
0.995261 (0.000990) with: {'max_features': 'sqrt', 'n_estimators': 10}
0.996110 (0.000754) with: {'max_features': 'sqrt', 'n_estimators': 100}
0.996081 (0.000853) with: {'max_features': 'sqrt', 'n_estimators': 1000}
0.995885 (0.000872) with: {'max_features': 'log2', 'n_estimators': 10}
0.996481 (0.000691) with: {'max_features': 'log2', 'n_estimators': 100}
0.996628 (0.000782) with: {'max_features': 'log2', 'n_estimators': 1000}
接下来，我想用模型绘制一个带有测试数据的混淆矩阵：
clf = RandomForestClassifier(max_features = 'log2', n_estimators=1000)
clf.fit(train_tf_w2v, y_train)
name = clf.__class__.__name__
expectation = y_test
test_prediction = clf.predict(test_tf_w2v)
acc = accuracy_score(expectation, test_prediction)
pre = precision_score(expectation, test_prediction)
rec = recall_score(expectation, test_prediction)
f1 = f1_score(expectation, test_prediction)
fig, ax = plt.subplots(1,2, figsize=(14,4))
plt.suptitle(f'{name} \n', fontsize = 18)
plt.subplots_adjust(top = 0.8)
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, ax=ax[0])
skplt.metrics.plot_confusion_matrix(expectation, test_prediction, normalize=True, ax = ax[1])
plt.show()
print(f"for the {name} we receive the following values:")
print("Accuracy: {:.3%}".format(acc))
print('Precision score: {:.3%}'.format(pre))
print('Recall score: {:.3%}'.format(rec))
print('F1 score: {:.3%}'.format(f1))
这输出：
对于 RandomForestClassifier，我们收到以下值： 准确度：57.974% 准确率：99.790% 召回率：15.983% F1分数：27.552%
哎哟...现在我觉得自己很傻。我发现哪里不对了。
在 train/test-split 之后，我将两个子集独立地发送到 W2Vvectorize() 函数。
train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)
从那里 W2Vvectorize() 函数基于两个独立的子集训练两个独立的 Word2Vec 模型。因此，当我将矢量化测试数据 test_tf_w2v 传递给我训练有素的 RandomForest 分类器时，为了检查测试集的准确性是否也正确，在训练有素的 RandomForest 分类器看来，测试集似乎在不同的语言。两个独立的 word2vec 模型只是以不同的方式向量化。
我按如下方式解决了这个问题：
def W2Vvectorize(X_train):
tokenize=X_train.apply(lambda x: x.split())
vector=[]
for sent in tqdm(tokenize):
sent_vec=np.zeros(100)
count =0
for word in sent:
if word in w2v_words:
vec = w2vec_model.wv[word]
sent_vec += vec
count += 1
if count != 0:
sent_vec /= count #normalize
vector.append(sent_vec)
return vector
并且 Word2Vec 训练与此分开：
x = train["training_tweet"]
y = train["label"]
X_train, X_test, y_train, y_test = train_test_split(x, y, test_size=0.2, stratify=train['label'])
print('X Train Shape = total * 0,8 =', X_train.shape)
print('y Train Shape = total * 0,8 =', y_train.shape)
print('X Test Shape = total * 0,2 =', X_test.shape)
print('y Test Shape = total * 0,2 =', y_test.shape) #
tokenize=X_train.apply(lambda x: x.split())
w2vec_model=gensim.models.Word2Vec(tokenize,min_count = 1, size = 100, window = 5, sg = 1)
w2vec_model.train(tokenize,total_examples= len(X_train), epochs=20)
w2v_words = list(w2vec_model.wv.vocab)
train_tf_w2v = W2Vvectorize(X_train)
test_tf_w2v = W2Vvectorize(X_test)
因此，Word2Vec 模型训练仅在训练数据上执行。然而，测试数据的矢量化必须使用完全相同的 Word2Vec 模型进行。
只是为了完整性：精度太高的原因是，我平衡了数据集，以便在最终训练集中均匀分布 classes。因此，我用更少的数据对 class 进行了上采样。这当然意味着之后数据集多次包含上采样数据。
如果您像我一样在上采样后拆分测试数据，那么您的训练数据很可能会包含来自上采样数据的元素。我画了一张图来解释：