保存模型以供以后预测 (OneVsRest)
Save model for later prediction (OneVsRest)
我想知道如何保存 OnevsRest classifier model 用于以后的预测。
我在保存它时遇到问题,因为这意味着还要保存矢量化器。我在这学到了.
这是我创建的模型:
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(strip_accents='unicode', analyzer='word', ngram_range=(1,3), norm='l2')
vectorizer.fit(train_text)
vectorizer.fit(test_text)
x_train = vectorizer.transform(train_text)
y_train = train.drop(labels = ['id','comment_text'], axis=1)
x_test = vectorizer.transform(test_text)
y_test = test.drop(labels = ['id','comment_text'], axis=1)
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
from sklearn.multiclass import OneVsRestClassifier
%%time
# Using pipeline for applying logistic regression and one vs rest classifier
LogReg_pipeline = Pipeline([
('clf', OneVsRestClassifier(LogisticRegression(solver='sag'), n_jobs=-1)),
])
for category in categories:
printmd('**Processing {} comments...**'.format(category))
# Training logistic regression model on train data
LogReg_pipeline.fit(x_train, train[category])
# calculating test accuracy
prediction = LogReg_pipeline.predict(x_test)
print('Test accuracy is {}'.format(accuracy_score(test[category], prediction)))
print("\n")
非常感谢任何帮助。
此致,
使用 joblib
您可以保存任何 Scikit-learn Pipeline
完整的所有元素,因此还包含拟合的 TfidfVectorizer
.
我在这里使用 Newsgroups20 数据集的前 200 个示例重写了您的示例:
from sklearn.datasets import fetch_20newsgroups
data = fetch_20newsgroups()
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
from sklearn.multiclass import OneVsRestClassifier
vectorizer = TfidfVectorizer(strip_accents='unicode', analyzer='word', ngram_range=(1,3), norm='l2')
x_train = data.data[:100]
y_train = data.target[:100]
x_test = data.data[100:200]
y_test = data.target[100:200]
# Using pipeline for applying logistic regression and one vs rest classifier
LogReg_pipeline = Pipeline([
('vectorizer', vectorizer),
('clf', OneVsRestClassifier(LogisticRegression(solver='sag',
class_weight='balanced'),
n_jobs=-1))
])
# Training logistic regression model on train data
LogReg_pipeline.fit(x_train, y_train)
在上面的代码中,您只需开始定义您的训练和测试数据,然后实例化您的 TfidfVectorizer
。然后,您定义包含矢量化器和 OVR 分类器的管道,并将其与训练数据相匹配。它将学习一次预测所有 类。
现在您只需使用 joblib
:
将整个拟合管道保存为单个预测变量
from joblib import dump, load
dump(LogReg_pipeline, 'LogReg_pipeline.joblib')
您的整个模型未以名称 'LogReg_pipeline.joblib' 保存到磁盘。您可以通过以下代码片段调用它并直接在原始数据上使用它:
clf = load('LogReg_pipeline.joblib')
clf.predict(x_test)
您将获得对原始文本的预测,因为管道会自动对其进行矢量化处理。
我想知道如何保存 OnevsRest classifier model 用于以后的预测。
我在保存它时遇到问题,因为这意味着还要保存矢量化器。我在这学到了
这是我创建的模型:
from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(strip_accents='unicode', analyzer='word', ngram_range=(1,3), norm='l2')
vectorizer.fit(train_text)
vectorizer.fit(test_text)
x_train = vectorizer.transform(train_text)
y_train = train.drop(labels = ['id','comment_text'], axis=1)
x_test = vectorizer.transform(test_text)
y_test = test.drop(labels = ['id','comment_text'], axis=1)
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
from sklearn.multiclass import OneVsRestClassifier
%%time
# Using pipeline for applying logistic regression and one vs rest classifier
LogReg_pipeline = Pipeline([
('clf', OneVsRestClassifier(LogisticRegression(solver='sag'), n_jobs=-1)),
])
for category in categories:
printmd('**Processing {} comments...**'.format(category))
# Training logistic regression model on train data
LogReg_pipeline.fit(x_train, train[category])
# calculating test accuracy
prediction = LogReg_pipeline.predict(x_test)
print('Test accuracy is {}'.format(accuracy_score(test[category], prediction)))
print("\n")
非常感谢任何帮助。
此致,
使用 joblib
您可以保存任何 Scikit-learn Pipeline
完整的所有元素,因此还包含拟合的 TfidfVectorizer
.
我在这里使用 Newsgroups20 数据集的前 200 个示例重写了您的示例:
from sklearn.datasets import fetch_20newsgroups
data = fetch_20newsgroups()
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score
from sklearn.multiclass import OneVsRestClassifier
vectorizer = TfidfVectorizer(strip_accents='unicode', analyzer='word', ngram_range=(1,3), norm='l2')
x_train = data.data[:100]
y_train = data.target[:100]
x_test = data.data[100:200]
y_test = data.target[100:200]
# Using pipeline for applying logistic regression and one vs rest classifier
LogReg_pipeline = Pipeline([
('vectorizer', vectorizer),
('clf', OneVsRestClassifier(LogisticRegression(solver='sag',
class_weight='balanced'),
n_jobs=-1))
])
# Training logistic regression model on train data
LogReg_pipeline.fit(x_train, y_train)
在上面的代码中,您只需开始定义您的训练和测试数据,然后实例化您的 TfidfVectorizer
。然后,您定义包含矢量化器和 OVR 分类器的管道,并将其与训练数据相匹配。它将学习一次预测所有 类。
现在您只需使用 joblib
:
from joblib import dump, load
dump(LogReg_pipeline, 'LogReg_pipeline.joblib')
您的整个模型未以名称 'LogReg_pipeline.joblib' 保存到磁盘。您可以通过以下代码片段调用它并直接在原始数据上使用它:
clf = load('LogReg_pipeline.joblib')
clf.predict(x_test)
您将获得对原始文本的预测,因为管道会自动对其进行矢量化处理。