带 tf-idf 变换的线性回归

Question

我有两个数据框，前者在列中包含 > 700 个预测变量，后者包含一列。前者用作预测变量（所有值都为 0 和 1，但由于稀疏性大部分为 0），第二个用作模型训练和测试的响应。第一个名称为 ser，第二个名称为 star.

我使用以下进行 tf-idf 转换

from sklearn.feature_extraction.text import TfidfTransformer
transformer = TfidfTransformer()

A = transformer.fit_transform(ser)

以下为部分print(A)

 (0, 302)   0.613133438876
 (0, 202)   0.789979358042
 (1, 556)   1.0
 (2, 556)   0.432375068194
 (2, 17)    0.901693850708
 (3, 556)   0.269567465847
 (3, 335)   0.671245025218
 (3, 256)   0.400099662956
 (3, 238)   0.562746618986
 (4, 556)   0.401348891903
 (4, 137)   0.915925251846
 (5, 641)   0.785485510985
 (5, 396)   0.618880046562
 (6, 317)   0.525163047715
 (6, 305)   0.851001629443
 ... (more are cut)

我用这个tf-idf转换没问题吧？由于我有以下内容，我收到错误消息，我将在 post.

的末尾 post

star = pd.DataFrame({"star": star})
data = pd.concat([ser, star], axis = 1)

from sklearn.linear_model import LinearRegression

D = LinearRegression()

Dfit = D.fit(ser, star, sample_weight = A)
Dpred = D.predict(ser)
Dscore = D.score(ser,star)
print(Dscore)

错误

Traceback (most recent call last):
File "categories_model.py", line 67, in <module>
Dfit = D.fit(ser, star, sample_weight = A)
File "/opt/conda/lib/python2.7/site-packages/sklearn/linear_model/base.py", line 434, in fit
sample_weight=sample_weight)
File "/opt/conda/lib/python2.7/site-packages/sklearn/linear_model/base.py", line 127, in center_data
X_mean = np.average(X, axis=0, weights=sample_weight)
File "/opt/conda/lib/python2.7/site-packages/numpy/lib/function_base.py", line 937, in average
"1D weights expected when shapes of a and weights differ.")
TypeError: 1D weights expected when shapes of a and weights differ.

谁能帮我理解所有这些以及如何改进代码？谢谢！！

Answer 1

错误来自放错变换后的矩阵。这解决了问题。

Dfit = D.fit(A, star)

带 tf-idf 变换的线性回归

Linear regression with tf-idf transformation

linear-regression

tf-idf

scikit-learn