ValueError: cannot use sparse input in 'SVC' trained on dense data

Question

我正在尝试运行我的分类器，但出现此错误

import pandas
import numpy as np
import pandas as pd
from sklearn import svm
from sklearn.svm import SVC
from sklearn import cross_validation
from sklearn.metrics import confusion_matrix
from sklearn.multiclass import OneVsOneClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_fscore_support as score
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer

dataset = pd.read_csv('all_topics_limpo.csv', encoding = 'utf-8')
data = pandas.get_dummies(dataset['verbatim_corrige'])
labels = dataset['label']

X_train, X_test, y_train, y_test = train_test_split(data, labels, test_size = 0.2, random_state = 0)

count_vector = CountVectorizer()
tfidf = TfidfTransformer()

classifier = OneVsOneClassifier(SVC(kernel = 'linear', random_state = 100))
#classifier = LogisticRegression()

train_counts = count_vector.fit_transform(X_train)
train_tfidf = tfidf.fit_transform(train_counts)
classifier.fit(X_train, y_train)

test_counts = count_vector.transform(X_test)
test_tfidf = tfidf.transform(test_counts)
predicted = classifier.predict(test_tfidf)

predicted = classifier.predict(X_test)

print("confusion matrix")
print(confusion_matrix(y_test, predicted, labels = labels))

print("F-score")
print(f1_score(y_test, predicted))
print(precision_score(y_test, predicted))
print(recall_score(y_test, predicted)) 

print("cross validation")
test_counts = count_vector.fit_transform(data)
test_tfidf = tfidf.fit_transform(test_counts)

scores = cross_validation.cross_val_score(classifier, test_tfidf, labels, cv = 10)
print(scores)
print("Accuracy: {} +/- {}".format(scores.mean(), scores.std() * 2))

我的输出错误：

ValueError: cannot use sparse input in 'SVC' trained on dense data

由于这个问题，我无法执行我的代码，而且我对正在发生的事情一无所知。

全部输出错误

Traceback (most recent call last):

File "classification.py", line 42, in predicted = classifier.predict(test_tfidf)

File "/usr/lib/python3/dist-packages/sklearn/multiclass.py", line 584, in predict Y = self.decision_function(X)

File "/usr/lib/python3/dist-packages/sklearn/multiclass.py", line 614, in decision_function for est, Xi in zip(self.estimators_, Xs)]).T

File "/usr/lib/python3/dist-packages/sklearn/multiclass.py", line 614, in for est, Xi in zip(self.estimators_, Xs)]).T

File "/usr/lib/python3/dist-packages/sklearn/svm/base.py", line 548, in predict y = super(BaseSVC, self).predict(X)

File "/usr/lib/python3/dist-packages/sklearn/svm/base.py", line 308, in predict X = self._validate_for_predict(X)

File "/usr/lib/python3/dist-packages/sklearn/svm/base.py", line 448, in _validate_for_predict % type(self).name)

ValueError: cannot use sparse input in 'SVC' trained on dense data

Answer 1

您收到此错误是因为您的训练和测试数据不是同一类型：当您在初始 X_train 集合中训练时：

classifier.fit(X_train, y_train)

您正在尝试从首先经过计数向量化和 tf-idf 转换的数据集中获取预测：

predicted = classifier.predict(test_tfidf)

令人困惑的是你为什么选择这样做，为什么你仍然计算 train_counts 和 train_tfidf（你似乎并没有在任何地方实际使用它们），以及为什么你也试图之后立即将 predicted 重新定义为 classifier.predict(X_test)。通常，将您的训练线更改为

classifier.fit(train_tfidf, y_train)

去掉你的第二个 predicted 定义应该没问题...

ValueError: cannot use sparse input in 'SVC' trained on dense data

ValueError: cannot use sparse input in 'SVC' trained on dense data

python

machine-learning

python-3.x

scikit-learn

text-classification