如何从一列数据帧中计算 tfidf 分数并提取具有最小分数阈值的单词

Question

我取了一列数据集，其中每一行都有文本形式的描述。我试图找到 tf-idf 大于某个值 n 的单词。但是代码给出了一个分数矩阵，我如何对分数进行排序和过滤并查看相应的单词。

tempdataFrame = wineData.loc[wineData.variety == 'Shiraz', 
'description'].reset_index()
tempdataFrame['description'] = tempdataFrame['description'].apply(lambda 
x: str.lower(x))

from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(analyzer='word', stop_words = 'english')
score = tfidf.fit_transform(tempDataFrame['description'])

Sample Data:
description
This tremendous 100% varietal wine hails from Oakville and was aged over 
three years in oak. Juicy red-cherry fruit and a compelling hint of caramel 
greet the palate, framed by elegant, fine tannins and a subtle minty tone in 
the background. Balanced and rewarding from start to finish, it has years 
ahead of it to develop further nuance. Enjoy 2022â€“2030.

Answer 1

由于没有完整的葡萄酒描述数据框列，您提供的样本数据被分成三个句子，以创建一个包含名为 'Description' 的一列和三行的数据框。然后将该列传递给 tf-idf 进行分析，并创建一个包含特征及其分数的新数据框。随后使用 pandas.

过滤结果

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

doc = ['This tremendous 100% varietal wine hails from Oakville and was aged over \
three years in oak.', 'Juicy red-cherry fruit and a compelling hint of caramel \
greet the palate, framed by elegant, fine tannins and a subtle minty tone in \
the background.', 'Balanced and rewarding from start to finish, it has years \
ahead of it to develop further nuance. Enjoy 2022â€“2030.']

df_1 = pd.DataFrame({'Description': doc})

tfidf = TfidfVectorizer(analyzer='word', stop_words = 'english')
score = tfidf.fit_transform(df_1['Description'])

# New data frame containing the tfidf features and their scores
df = pd.DataFrame(score.toarray(), columns=tfidf.get_feature_names())

# Filter the tokens with tfidf score greater than 0.3
tokens_above_threshold = df.max()[df.max() > 0.3].sort_values(ascending=False)

tokens_above_threshold
Out[29]: 
wine          0.341426
oak           0.341426
aged          0.341426
varietal      0.341426
hails         0.341426
100           0.341426
oakville      0.341426
tremendous    0.341426
nuance        0.307461
rewarding     0.307461
start         0.307461
enjoy         0.307461
develop       0.307461
balanced      0.307461
ahead         0.307461
2030          0.307461
2022â         0.307461
finish        0.307461

如何从一列数据帧中计算 tfidf 分数并提取具有最小分数阈值的单词

How to calculate tfidf score from a column of dataframe and extract words with a minimum score threshold

tf-idf

pandas