尝试在包含 Pandas 数据帧列（包含字符串）的 TFidfVectorizer 上应用 'fit_transform()' 时出现内存错误

Question

我正在尝试类似的操作，如图所示。我首先从包含 2405 行的 CSV 文件中读取两列，格式如下： “1995”和清洁例如["this"、"is, "exemplar"、"document"、"contents"]，两列都使用字符串作为数据类型。

    df = pandas.read_csv("ukgovClean.csv", encoding='utf-8', usecols=[0,2])

我已经预先清理了数据，下面显示了前4行的格式：

     [IN] df.head()

    [OUT]   Year    cleaned
         0  1909    acquaint hous receiv follow letter clerk crown...
         1  1909    ask secretari state war whether issu statement...
         2  1909    i beg present petit sign upward motor car driv...
         3  1909    i desir ask secretari state war second lieuten...
         4  1909    ask secretari state war whether would introduc...

    [IN] df['cleaned'].head()

   [OUT] 0    acquaint hous receiv follow letter clerk crown...
         1    ask secretari state war whether issu statement...
         2    i beg present petit sign upward motor car driv...
         3    i desir ask secretari state war second lieuten...
         4    ask secretari state war whether would introduc...
         Name: cleaned, dtype: object

然后我初始化 TfidfVectorizer：

    [IN] v = TfidfVectorizer(decode_error='replace', encoding='utf-8')

在此之后，调用以下行结果：

    [IN] x = v.fit_transform(df['cleaned'])
   [OUT] ValueError: np.nan is an invalid document, expected byte or unicode string.

我使用上述中的解决方案克服了这个问题：

    [IN] x = v.fit_transform(df['cleaned'].values.astype('U'))

但是，这导致内存错误 (Full Traceback)。

我试图使用 Pickle 查找存储以规避大容量内存的使用，但我不确定在这种情况下如何过滤它。任何提示将不胜感激，感谢阅读。

[更新]

@pittsburgh137 posted a solution to a similar problem involving fitting data , in which the training data is generated using pandas.get_dummies()。我对此所做的是：

    [IN] train_X = pandas.get_dummies(df['cleaned'])
    [IN] train_X.shape
   [OUT] (2405, 2380)

    [IN] x = v.fit_transform(train_X)
    [IN] type(x)
   [OUT] scipy.sparse.csr.csr_matrix

我想我应该在看到我可以用这个开发做些什么的同时更新所有读者。如果此方法有任何预测的缺陷，我很乐意听到它们。

Answer 1

我认为是 dtype('<Unn') 的转换可能会给您带来麻烦。在相对基础上检查数组的大小，仅使用前几个文档加上 NaN:

>>> df['cleaned'].values
array(['acquaint hous receiv follow letter clerk crown',
       'ask secretari state war whether issu statement',
       'i beg present petit sign upward motor car driv',
       'i desir ask secretari state war second lieuten',
       'ask secretari state war whether would introduc', nan],
      dtype=object)

>>> df['cleaned'].values.astype('U').nbytes
1104

>>> df['cleaned'].values.nbytes
48

似乎首先删除 NaN 值是有意义的 (df.dropna(inplace=True))。然后，调用 v.fit_transform(df['cleaned'].tolist()).

应该非常有效

尝试在包含 Pandas 数据帧列（包含字符串）的 TFidfVectorizer 上应用 'fit_transform()' 时出现内存错误

Memory Error when attempting to apply 'fit_transform()' on TFidfVectorizer containing Pandas Dataframe column (containing strings)

python

csv

tf-idf

pandas

scikit-learn