Pandas 数据帧返回错误形状的 CountVectorizer

Question

import numpy as np
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import (f1_score,precision_score,recall_score)
ifile=open("train_pos.txt")
rows = []
for ln in ifile:
    rows.append({'text': ln, 'class': 1})
ifile.close()
data_frame = pd.DataFrame(rows)
data_frame

此代码输出：

text    class
0   Coffee is great and I live close so it's conve...   1
1   I love this place for its coffeeshop feel with...   1
2   I've come here now a couple of times and I lov...   1
3   Nice vibes, just ok food and not too warm serv...   1
4   Not a big breakfast person but this place has ...   1
... ... ...
74241   Henry the bartender makes a stop in at Bijans ...   1
74242   I love the ambiance at Bijan, especially the w...   1
74243   Popped in for Happy Hour, on a hot and stormy ...   1
74244   Update: Â Bijan's came on the scene as a great...   1
74245   The nearly day-glo lime green walls, harsh flu...   1
74246 rows × 2 columns

我正在尝试使用 countvectorizer 以 pandas data_frame 作为输入来执行特征提取。

为此，我编写了代码

count_vect = CountVectorizer() 
X_train_counts = count_vect.fit_transform(data_frame.text)
X_train_counts.shape

问题是当我运行上面的代码时，它给了我错误的形状。形状正在输出 (74246, 61803) 但它应该输出 (74246, 2)。当我运行时它给出了正确的输出 data_frame.shape

有谁知道为什么会发生这种情况以及如何解决它？

非常感谢任何帮助！

Answer 1

您可能会混淆 fit_transform() 和 fit()。 fit_transform() 学习词汇字典，然后将其转换为文档-术语矩阵。所以你得到的是矩阵而不是字典。 fit_transform 与运行 fit 后跟变换相同。因此，如果您要查找字典，只需使用 fit()

Pandas 数据帧返回错误形状的 CountVectorizer

CountVectorizer with Pandas dataframe returning wrong shape

python

pandas

reshape

data-mining

countvectorizer