是在使用逻辑回归进行情感分析时获得积极或消极程度的一种方法
Is the a way of getting the degree of positiveness or negativeness when using Logistic Regression for sentiment analysis
我一直在关注使用逻辑回归进行情感分析的示例,其中预测结果仅给出 1 或 0 分别给出正面或负面情绪。
我的挑战是我想将给定的用户输入分类为四个 类(非常好、好、一般、差)之一,但我每次的预测结果都是 1 或 0。
下面是我目前的代码示例
from sklearn.feature_extraction.text import CountVectorizer
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
from sklearn.metrics import classification_report
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_files
from sklearn.model_selection import GridSearchCV
import numpy as np
import mglearn
import matplotlib.pyplot as plt
# import warnings filter
from warnings import simplefilter
# ignore all future warnings
#simplefilter(action='ignore', category=FutureWarning)
# Get the dataset from http://ai.stanford.edu/~amaas/data/sentiment/
reviews_train = load_files("aclImdb/train/")
text_train, y_train = reviews_train.data, reviews_train.target
print("")
print("Number of documents in train data: {}".format(len(text_train)))
print("")
print("Samples per class (train): {}".format(np.bincount(y_train)))
print("")
reviews_test = load_files("aclImdb/test/")
text_test, y_test = reviews_test.data, reviews_test.target
print("Number of documents in test data: {}".format(len(text_test)))
print("")
print("Samples per class (test): {}".format(np.bincount(y_test)))
print("")
vect = CountVectorizer(stop_words="english", analyzer='word',
ngram_range=(1, 1), max_df=1.0, min_df=1,
max_features=None)
X_train = vect.fit(text_train).transform(text_train)
X_test = vect.transform(text_test)
print("Vocabulary size: {}".format(len(vect.vocabulary_)))
print("")
print("X_train:\n{}".format(repr(X_train)))
print("X_test: \n{}".format(repr(X_test)))
feature_names = vect.get_feature_names()
print("Number of features: {}".format(len(feature_names)))
print("")
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10]}
grid =
GridSearchCV(LogisticRegression(penalty='l1',dual=False,max_iter=110,
solver='liblinear'), param_grid, cv=5)
grid.fit(X_train, y_train)
print("Best cross-validation score: {:.2f}".format(grid.best_score_))
print("Best parameters: ", grid.best_params_)
print("Best estimator: ", grid.best_estimator_)
lr = grid.best_estimator_
lr.predict(X_test)
print("Best Estimator Score: {:.2f}".format(lr.score(X_test, y_test)))
print("")
#creating an empty list for getting overall sentiment
lst = []
# number of elemetns as input
print("")
n = int(input("Enter number of rounds : "))
# iterating till the range
for i in range(0, n):
temp =[]
ele = input("\n Please Enter a sentence to get a sentiment Evaluation.
\n\n")
temp.append(ele)
print("")
print("Review prediction: {}". format(lr.predict(vect.transform(temp))))
print("")
lst.append(ele) # adding the element
print(lst)
print("")
print("Overal prediction: {}". format(lr.predict(vect.transform(lst))))
print("")
我想获得一些介于 -0 到 1 之间的值,就像您使用 Vader SentimentIntensityAnalyzer 的 polarity_scores 一样。
这是我想使用 SentimentIntensityAnalyzer 的 polarity_scores.
实现的代码示例
# import SentimentIntensityAnalyzer class
# from vaderSentiment.vaderSentiment module.
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
# function to print sentiments
# of the sentence.
def sentiment_scores(sentence):
# Create a SentimentIntensityAnalyzer object.
sid_obj = SentimentIntensityAnalyzer()
# polarity_scores method of SentimentIntensityAnalyzer
# oject gives a sentiment dictionary.
# which contains pos, neg, neu, and compound scores.
sentiment_dict = sid_obj.polarity_scores(sentence)
print("")
print("\n Overall sentiment dictionary is : ", sentiment_dict," \n")
print("sentence was rated as: ", sentiment_dict['neg']*100, "% Negative
\n")
print("sentence was rated as: ", sentiment_dict['neu']*100, "% Neutral
\n")
print("sentence was rated as: ", sentiment_dict['pos']*100, "% Positive
\n")
print("Sentence Overall Rated As: ", end = " ")
# decide sentiment as positive, negative and neutral
if sentiment_dict['compound'] >= 0.5:
print("Exellent \n")
elif sentiment_dict['compound'] > 0 and sentiment_dict['compound'] <0.5:
print("Very Good \n")
elif sentiment_dict['compound'] == 0:
print("Good \n")
elif sentiment_dict['compound'] <= -0.5:
print("Average \n")
elif sentiment_dict['compound'] > -0.5 and sentiment_dict['compound']<0:
print("Poor \n")
# Driver code
if __name__ == "__main__" :
while True:
# print("")
sentence= []
sentence = input("\n Please enter a sentence to get a sentimet
evaluation. Enter exit to end progam \n")
if sentence == "exit":
print("\n Program End...........\n")
print("")
break
else:
sentiment_scores(sentence)
你有几个选择。
1:根据示例的消极或积极程度,用多个 类 标记您的初始训练数据,而不仅仅是 0 或 1,然后执行 multi-class 分类。
2:由于 1 可能无法实现,请尝试使用 predict_proba(X)
、predict_log_proba(X)
和 decision_function(X)
方法进行试验,并使用这些方法的结果将您的输出放入 4 类 根据一些 hard-coded 阈值。我建议使用 predict_proba
,因为这些数字可以直接解释为概率,并且是逻辑回归相对于其他方法的主要优势之一。例如,假设第 1(不是第 0)列是 "positive" 分类
probs = lr.predict_proba(X_test)
labels = np.repeat("very_good", len(probs))
labels[probs[:, 1] < 0.75] = "good"
labels[probs[:, 1] < 0.5] = "average"
labels[probs[:, 1] < 0.25] = "poor"
我一直在关注使用逻辑回归进行情感分析的示例,其中预测结果仅给出 1 或 0 分别给出正面或负面情绪。
我的挑战是我想将给定的用户输入分类为四个 类(非常好、好、一般、差)之一,但我每次的预测结果都是 1 或 0。
下面是我目前的代码示例
from sklearn.feature_extraction.text import CountVectorizer
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
from sklearn.metrics import classification_report
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_files
from sklearn.model_selection import GridSearchCV
import numpy as np
import mglearn
import matplotlib.pyplot as plt
# import warnings filter
from warnings import simplefilter
# ignore all future warnings
#simplefilter(action='ignore', category=FutureWarning)
# Get the dataset from http://ai.stanford.edu/~amaas/data/sentiment/
reviews_train = load_files("aclImdb/train/")
text_train, y_train = reviews_train.data, reviews_train.target
print("")
print("Number of documents in train data: {}".format(len(text_train)))
print("")
print("Samples per class (train): {}".format(np.bincount(y_train)))
print("")
reviews_test = load_files("aclImdb/test/")
text_test, y_test = reviews_test.data, reviews_test.target
print("Number of documents in test data: {}".format(len(text_test)))
print("")
print("Samples per class (test): {}".format(np.bincount(y_test)))
print("")
vect = CountVectorizer(stop_words="english", analyzer='word',
ngram_range=(1, 1), max_df=1.0, min_df=1,
max_features=None)
X_train = vect.fit(text_train).transform(text_train)
X_test = vect.transform(text_test)
print("Vocabulary size: {}".format(len(vect.vocabulary_)))
print("")
print("X_train:\n{}".format(repr(X_train)))
print("X_test: \n{}".format(repr(X_test)))
feature_names = vect.get_feature_names()
print("Number of features: {}".format(len(feature_names)))
print("")
param_grid = {'C': [0.001, 0.01, 0.1, 1, 10]}
grid =
GridSearchCV(LogisticRegression(penalty='l1',dual=False,max_iter=110,
solver='liblinear'), param_grid, cv=5)
grid.fit(X_train, y_train)
print("Best cross-validation score: {:.2f}".format(grid.best_score_))
print("Best parameters: ", grid.best_params_)
print("Best estimator: ", grid.best_estimator_)
lr = grid.best_estimator_
lr.predict(X_test)
print("Best Estimator Score: {:.2f}".format(lr.score(X_test, y_test)))
print("")
#creating an empty list for getting overall sentiment
lst = []
# number of elemetns as input
print("")
n = int(input("Enter number of rounds : "))
# iterating till the range
for i in range(0, n):
temp =[]
ele = input("\n Please Enter a sentence to get a sentiment Evaluation.
\n\n")
temp.append(ele)
print("")
print("Review prediction: {}". format(lr.predict(vect.transform(temp))))
print("")
lst.append(ele) # adding the element
print(lst)
print("")
print("Overal prediction: {}". format(lr.predict(vect.transform(lst))))
print("")
我想获得一些介于 -0 到 1 之间的值,就像您使用 Vader SentimentIntensityAnalyzer 的 polarity_scores 一样。
这是我想使用 SentimentIntensityAnalyzer 的 polarity_scores.
实现的代码示例# import SentimentIntensityAnalyzer class
# from vaderSentiment.vaderSentiment module.
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
# function to print sentiments
# of the sentence.
def sentiment_scores(sentence):
# Create a SentimentIntensityAnalyzer object.
sid_obj = SentimentIntensityAnalyzer()
# polarity_scores method of SentimentIntensityAnalyzer
# oject gives a sentiment dictionary.
# which contains pos, neg, neu, and compound scores.
sentiment_dict = sid_obj.polarity_scores(sentence)
print("")
print("\n Overall sentiment dictionary is : ", sentiment_dict," \n")
print("sentence was rated as: ", sentiment_dict['neg']*100, "% Negative
\n")
print("sentence was rated as: ", sentiment_dict['neu']*100, "% Neutral
\n")
print("sentence was rated as: ", sentiment_dict['pos']*100, "% Positive
\n")
print("Sentence Overall Rated As: ", end = " ")
# decide sentiment as positive, negative and neutral
if sentiment_dict['compound'] >= 0.5:
print("Exellent \n")
elif sentiment_dict['compound'] > 0 and sentiment_dict['compound'] <0.5:
print("Very Good \n")
elif sentiment_dict['compound'] == 0:
print("Good \n")
elif sentiment_dict['compound'] <= -0.5:
print("Average \n")
elif sentiment_dict['compound'] > -0.5 and sentiment_dict['compound']<0:
print("Poor \n")
# Driver code
if __name__ == "__main__" :
while True:
# print("")
sentence= []
sentence = input("\n Please enter a sentence to get a sentimet
evaluation. Enter exit to end progam \n")
if sentence == "exit":
print("\n Program End...........\n")
print("")
break
else:
sentiment_scores(sentence)
你有几个选择。
1:根据示例的消极或积极程度,用多个 类 标记您的初始训练数据,而不仅仅是 0 或 1,然后执行 multi-class 分类。
2:由于 1 可能无法实现,请尝试使用 predict_proba(X)
、predict_log_proba(X)
和 decision_function(X)
方法进行试验,并使用这些方法的结果将您的输出放入 4 类 根据一些 hard-coded 阈值。我建议使用 predict_proba
,因为这些数字可以直接解释为概率,并且是逻辑回归相对于其他方法的主要优势之一。例如,假设第 1(不是第 0)列是 "positive" 分类
probs = lr.predict_proba(X_test)
labels = np.repeat("very_good", len(probs))
labels[probs[:, 1] < 0.75] = "good"
labels[probs[:, 1] < 0.5] = "average"
labels[probs[:, 1] < 0.25] = "poor"