使用 Python 在多个列表中高效实现字数统计
Efficient implementation of words count in the several lists using Python
我有以下格式的评论列表:
Comments=[['hello world'], ['would', 'hard', 'press'],['find', 'place', 'less'']]
wordset={'hello','world','hard','would','press','find','place','less'}
我希望 table 或数据框以词集作为索引,并且 Comments
中每个评论的个体计数
我使用以下代码实现了所需的数据帧。这是很花时间的,我正在寻找一个有效的实施。由于语料库很大,这对我们排序算法的效率有很大的影响。
result=pd.DataFrame()
for comment in Comments:
worddict_terms=dict.fromkeys(wordset,0)
for items in comment:
worddict_terms[items]+=1
df_comment=pd.DataFrame.from_dict([worddict_terms])
frames=[result,df_comment]
result = pd.concat(frames)
Comments_raw_terms=result.transpose()
我们期望的结果是:
0 1 2
hello 1 0 0
world 1 0 0
would 0 1 0
press 0 1 0
find 0 0 1
place 0 0 1
less 0 0 1
hard 0 1 0
试试这个方法:
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer()
text = pd.Series(Comments).str.join(' ')
X = vect.fit_transform(text)
r = pd.DataFrame(X.toarray(), columns=vect.get_feature_names())
结果:
In [49]: r
Out[49]:
find hard hello less place press world would
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0 1
2 1 0 0 1 1 0 0 0
In [50]: r.T
Out[50]:
0 1 2
find 0 0 1
hard 0 1 0
hello 1 0 0
less 0 0 1
place 0 0 1
press 0 1 0
world 1 0 0
would 0 1 0
纯Pandas解决方案:
In [61]: pd.get_dummies(text.str.split(expand=True), prefix_sep='', prefix='')
Out[61]:
find hello would hard place world less press
0 0 1 0 0 0 1 0 0
1 0 0 1 1 0 0 0 1
2 1 0 0 0 1 0 1 0
我认为您的嵌套 for 循环 正在增加复杂性。我正在编写代码,用单个映射函数 替换 2 for 循环。我只写代码到评论中的每个评论的部分,你得到 "Hello" 和 "World" 的 count_dictionary。您,请使用 pandas.
复制制作 table 的剩余代码
from collections import Counter
import funcy
from funcy import project
def fun(comment):
wordset={'hello','world'}
temp_dict_comment = Counter(comment)
temp_dict_comment = dict(temp_dict_comment)
final_dict = project(temp_dict_comment,wordset)
print final_dict
Comments=[['hello', 'world'], ['would', 'hard', 'press'],['find', 'place', 'less', 'excitingit', 'wors', 'watch', 'paint', 'dri']]
map(fun,Comments)
这应该有所帮助,因为它只包含 单个映射而不是 2 个 for 循环。
我有以下格式的评论列表:
Comments=[['hello world'], ['would', 'hard', 'press'],['find', 'place', 'less'']]
wordset={'hello','world','hard','would','press','find','place','less'}
我希望 table 或数据框以词集作为索引,并且 Comments
中每个评论的个体计数我使用以下代码实现了所需的数据帧。这是很花时间的,我正在寻找一个有效的实施。由于语料库很大,这对我们排序算法的效率有很大的影响。
result=pd.DataFrame()
for comment in Comments:
worddict_terms=dict.fromkeys(wordset,0)
for items in comment:
worddict_terms[items]+=1
df_comment=pd.DataFrame.from_dict([worddict_terms])
frames=[result,df_comment]
result = pd.concat(frames)
Comments_raw_terms=result.transpose()
我们期望的结果是:
0 1 2
hello 1 0 0
world 1 0 0
would 0 1 0
press 0 1 0
find 0 0 1
place 0 0 1
less 0 0 1
hard 0 1 0
试试这个方法:
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
vect = CountVectorizer()
text = pd.Series(Comments).str.join(' ')
X = vect.fit_transform(text)
r = pd.DataFrame(X.toarray(), columns=vect.get_feature_names())
结果:
In [49]: r
Out[49]:
find hard hello less place press world would
0 0 0 1 0 0 0 1 0
1 0 1 0 0 0 1 0 1
2 1 0 0 1 1 0 0 0
In [50]: r.T
Out[50]:
0 1 2
find 0 0 1
hard 0 1 0
hello 1 0 0
less 0 0 1
place 0 0 1
press 0 1 0
world 1 0 0
would 0 1 0
纯Pandas解决方案:
In [61]: pd.get_dummies(text.str.split(expand=True), prefix_sep='', prefix='')
Out[61]:
find hello would hard place world less press
0 0 1 0 0 0 1 0 0
1 0 0 1 1 0 0 0 1
2 1 0 0 0 1 0 1 0
我认为您的嵌套 for 循环 正在增加复杂性。我正在编写代码,用单个映射函数 替换 2 for 循环。我只写代码到评论中的每个评论的部分,你得到 "Hello" 和 "World" 的 count_dictionary。您,请使用 pandas.
复制制作 table 的剩余代码from collections import Counter
import funcy
from funcy import project
def fun(comment):
wordset={'hello','world'}
temp_dict_comment = Counter(comment)
temp_dict_comment = dict(temp_dict_comment)
final_dict = project(temp_dict_comment,wordset)
print final_dict
Comments=[['hello', 'world'], ['would', 'hard', 'press'],['find', 'place', 'less', 'excitingit', 'wors', 'watch', 'paint', 'dri']]
map(fun,Comments)
这应该有所帮助,因为它只包含 单个映射而不是 2 个 for 循环。