即使使用 csr_matrix，如何使用 BinaryRelevance 修复 ArrayMemoryError？

Question

预测有毒评论

import skmultilearn, sys
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from scipy.sparse import csr_matrix, issparse
from sklearn.naive_bayes import MultinomialNB
from skmultilearn.problem_transform import BinaryRelevance

data_frame = pd.read_csv('data/train.csv')
corpus = data_frame['comment_text']
tfidf = TfidfVectorizer()
Xfeatures = csr_matrix(tfidf.fit_transform(corpus))
y = csr_matrix(data_frame[['toxic','severe_toxic','obscene','threat','insult','identity_hate']])
binary_rel_clf = BinaryRelevance(MultinomialNB())
binary_rel_clf.fit(Xfeatures,y)
predict_text = ['fuck die shit moron suck']
X_predict = tfidf.transform(predict_text)
br_prediction = binary_rel_clf.predict(X_predict)
br_prediction = br_prediction.toarray().astype(bool)
predictions = [y.columns.values[prediction].tolist() for prediction in br_prediction]
print(predictions)

但是，我得到了这个错误：

Traceback (most recent call last):
  File "...\multi_label_toxic.py", line 15, in <module>
    binary_rel_clf.fit(Xfeatures,y)
  File "...\problem_transform\br.py", line 161, in fit
    classifier.fit(self._ensure_input_format(
  File "...\base\base.py", line 86, in _ensure_input_format
    return X.toarray()
  File "...\scipy\sparse\compressed.py", line 1031, in toarray
    out = self._process_toarray_args(order, out)
  File "...\scipy\sparse\base.py", line 1202, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 226. GiB for an array with shape (159571, 189775) and data type float64

即使尝试传递参数“require_dense=False”，我也会收到另一个错误：

Traceback (most recent call last):
  File "...\multi_label_toxic.py", line 15, in <module>
    binary_rel_clf.fit(Xfeatures,y)
  File "...\skmultilearn\problem_transform\br.py", line 161, in fit
    classifier.fit(self._ensure_input_format(
  File "...\sklearn\naive_bayes.py", line 612, in fit
    X, y = self._check_X_y(X, y)
  File "...\sklearn\naive_bayes.py", line 477, in _check_X_y
    return self._validate_data(X, y, accept_sparse='csr')
  File "...\sklearn\base.py", line 433, in _validate_data
    X, y = check_X_y(X, y, **check_params)
  File "...\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "...\sklearn\utils\validation.py", line 826, in check_X_y
    y = column_or_1d(y, warn=True)
  File "...\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "...\sklearn\utils\validation.py", line 864, in column_or_1d
    raise ValueError(
ValueError: y should be a 1d array, got an array of shape () instead.

我该如何解决这个问题并使用整个模型进行训练？

Answer 1

您似乎错误地指定了 required_dense 参数。您需要 required_dense=[False, True] 才能以稀疏格式指定 X 值而不是 y 值。在倒数第二行 (predictions = ...) 中，您需要在将其转换为矩阵之前使用 y ，以便您可以访问列名。以下代码应该可以工作。

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from scipy.sparse import csr_matrix, issparse
from sklearn.naive_bayes import MultinomialNB
from skmultilearn.problem_transform import BinaryRelevance
import numpy as np

data_frame = pd.read_csv('data/train.csv')
corpus = data_frame['comment_text']
tfidf = TfidfVectorizer()
Xfeatures = csr_matrix(tfidf.fit_transform(corpus))
cats = data_frame[['toxic','severe_toxic','obscene','threat','insult','identity_hate']]
y = csr_matrix(cats)
binary_rel_clf = BinaryRelevance(MultinomialNB(), require_dense = [False, True])
binary_rel_clf.fit(Xfeatures, y) # y[:,0].toarray().reshape(-1)
predict_text = ['fuck die shit moron suck']
X_predict = tfidf.transform(predict_text)
br_prediction = binary_rel_clf.predict(X_predict)
br_prediction = br_prediction.toarray().astype(bool)
predictions = [cats.columns[prediction].tolist() for prediction in br_prediction]
print(predictions)

输出：

[['toxic', 'obscene', 'insult']]

即使使用 csr_matrix，如何使用 BinaryRelevance 修复 ArrayMemoryError？

How to fix ArrayMemoryError using BinaryRelevance even using csr_matrix?

python

sparse-matrix

multilabel-classification

skmultilearn