检查同一列中是否有相似的字符串
Check if there is a similar string in the same column
我有这样一个数据框,
df
col1 col2
A 'the value is zero'
B 'this is a cat'
C 'the value is one'
D 'nothing is here'
E 'the colour is blue'
F 'this is dog'
G 'empty sequence'
H 'the colour is red'
I 'the colour is green' 1
现在我想要类似的字符串标记为 1,其他标记为零,所以最终的数据框应该是这样的,
col1 col2 col1
A 'the value is zero' 1
B 'this is a cat' 1
C 'the value is one' 1
D 'nothing is here' 0
E 'the colour is blue' 1
F 'this is dog' 1
G 'empty sequence' 0
H 'the colour is red' 1
I 'the colour is green' 1
可以使用 SequenceMatcher(SequenceMatcher(None, s1, s2).ratio()) 函数获得 0 和 1,通过一些阈值我们可以将其设为零或一。
但是如果我使用for循环来寻找彼此之间的相似性,那么它会花费更长的时间来执行。寻找一些 pandas shortcuts/pythonic 方法来有效地做到这一点。
类似于is it possible to do fuzzy match merge with python pandas?,
我们可以使用 difflib
并通过查看 difflib.get_close_matches
:
返回的列表的长度来检查我们是否找到超过 1 个相似的字符串(排除它自己的)
import difflib
df['col1'] = [(len(difflib.get_close_matches(x, df['col2'], cutoff=0.7))>1)*1
for x in df['col2']]
print(df)
col1 col2
0 1 'the value is zero'
1 1 'this is a cat'
2 1 'the value is one'
3 0 'nothing is here'
4 1 'the colour is blue'
5 1 'this is dog'
6 0 'empty sequence'
7 1 'the colour is red'
8 1 'the colour is green'
基于fuzzy matching
的相似度矩阵
如果字符串相似,也可能有兴趣获得一个相似矩阵,将旋转列中的所有值设置为 1
。为此,我们可以像上面一样进行,但保留整个列表,将其分解并使用 pd.crosstab
:
旋转生成的数据帧
df['sim'] = [difflib.get_close_matches(x, df['col2'], cutoff=0.7) for x in df['col2']]
sim_df = df.explode('sim')
pd.crosstab(sim_df.col2, sim_df.sim)
sim empty sequence nothing is here the colour is blue... the value is zero this is a cat this is dog
col2
empty sequence 1 0 0 ... 0 0 0
nothing is here 0 1 0 ... 0 0 0
the colour is blue 0 0 1 ... 0 0 0
the colour is green 0 0 1 ... 0 0 0
the colour is red 0 0 1 ... 0 0 0
the value is one 0 0 0 ... 1 0 0
the value is zero 0 0 0 ... 1 0 0
this is a cat 0 0 0 ... 0 1 1
this is dog 0 0 0 ... 0 1 1
我有这样一个数据框,
df
col1 col2
A 'the value is zero'
B 'this is a cat'
C 'the value is one'
D 'nothing is here'
E 'the colour is blue'
F 'this is dog'
G 'empty sequence'
H 'the colour is red'
I 'the colour is green' 1
现在我想要类似的字符串标记为 1,其他标记为零,所以最终的数据框应该是这样的,
col1 col2 col1
A 'the value is zero' 1
B 'this is a cat' 1
C 'the value is one' 1
D 'nothing is here' 0
E 'the colour is blue' 1
F 'this is dog' 1
G 'empty sequence' 0
H 'the colour is red' 1
I 'the colour is green' 1
可以使用 SequenceMatcher(SequenceMatcher(None, s1, s2).ratio()) 函数获得 0 和 1,通过一些阈值我们可以将其设为零或一。
但是如果我使用for循环来寻找彼此之间的相似性,那么它会花费更长的时间来执行。寻找一些 pandas shortcuts/pythonic 方法来有效地做到这一点。
类似于is it possible to do fuzzy match merge with python pandas?,
我们可以使用 difflib
并通过查看 difflib.get_close_matches
:
import difflib
df['col1'] = [(len(difflib.get_close_matches(x, df['col2'], cutoff=0.7))>1)*1
for x in df['col2']]
print(df)
col1 col2
0 1 'the value is zero'
1 1 'this is a cat'
2 1 'the value is one'
3 0 'nothing is here'
4 1 'the colour is blue'
5 1 'this is dog'
6 0 'empty sequence'
7 1 'the colour is red'
8 1 'the colour is green'
基于fuzzy matching
的相似度矩阵如果字符串相似,也可能有兴趣获得一个相似矩阵,将旋转列中的所有值设置为 1
。为此,我们可以像上面一样进行,但保留整个列表,将其分解并使用 pd.crosstab
:
df['sim'] = [difflib.get_close_matches(x, df['col2'], cutoff=0.7) for x in df['col2']]
sim_df = df.explode('sim')
pd.crosstab(sim_df.col2, sim_df.sim)
sim empty sequence nothing is here the colour is blue... the value is zero this is a cat this is dog
col2
empty sequence 1 0 0 ... 0 0 0
nothing is here 0 1 0 ... 0 0 0
the colour is blue 0 0 1 ... 0 0 0
the colour is green 0 0 1 ... 0 0 0
the colour is red 0 0 1 ... 0 0 0
the value is one 0 0 0 ... 1 0 0
the value is zero 0 0 0 ... 1 0 0
this is a cat 0 0 0 ... 0 1 1
this is dog 0 0 0 ... 0 1 1