Python 列表中所有可能对的余弦相似度

Question

我学习 Python 很慢，想知道是否需要一些帮助。我有一个 ips 列表，occurrence_id 和名为 info_list:

的矢量

('188.74.64.243', '1', ['0, 1, 1, 0'])
('99.229.98.18',  '1', ['0, 1, 1, 1'])
('86.41.253.102', '1', ['1, 1, 1, 1'])
('188.74.64.243', '2', ['0, 1, 1, 0'])
('99.229.98.18',  '2', ['0, 1, 1, 1'])
('86.41.253.102', '2', ['1, 1, 1, 1'])

我想计算余弦相似度。我有以下内容：

def cosine_similarity(v1,v2):
    sumxx, sumxy, sumyy = 0, 0, 0
    for i in range(len(v1)):
        x = v1[i]; y = v2[i]
        sumxx += x*x
        sumyy += y*y
        sumxy += x*y
    return sumxy/math.sqrt(sumxx*sumyy)

v1 = [0, 1, 1, 0]
v2 = [1, 1, 1, 1]
print(v1, v2, cosine_similarity(v1,v2))

这在声明 v1 和 v2 时效果很好。我的问题是我有点陷入困境，似乎无法拼凑我的问题。我希望得到一点帮助。

我需要遍历 info_list，考虑到具有相同 [=24] 的每对 ips =]occurrence_id来计算cosine_similarity.

输出的一个例子是这样的列表：

    ('188.74.64.243', '99.229.98.18', '1', ['0, 1, 1, 0'],['0, 1, 1, 1'], 0.82 )
    ('188.74.64.243', '86.41.253.102', '1', ['0, 1, 1, 0'],['1, 1, 1, 1'], 0.70 )
    ('86.41.253.102', '99.229.98.18', '1', ['0, 1, 1, 1'],['1, 1, 1, 1'], 0.87 )

Answer 1

您可以使用 Python 的 groupby and combinations 功能，如下所示：

from itertools import groupby, combinations
import math

def cosine_similarity(v1,v2):
    sumxx, sumxy, sumyy = 0, 0, 0
    for i in range(len(v1)):
        x = v1[i]; y = v2[i]
        sumxx += x*x
        sumyy += y*y
        sumxy += x*y
    return sumxy/math.sqrt(sumxx * sumyy)

info_list = [
    ('188.74.64.243', '1', [0, 1, 1, 0]),
    ('99.229.98.18',  '1', [0, 1, 1, 1]),
    ('86.41.253.102', '1', [1, 1, 1, 1]),
    ('188.74.64.243', '2', [0, 1, 1, 0]),
    ('99.229.98.18',  '2', [0, 1, 1, 1]),
    ('86.41.253.102', '2', [1, 1, 1, 1]),
    ]

for k, g in groupby(info_list, key=lambda x: x[1]):
    for x, y in combinations(g, 2):
        print (x[0], y[0], x[1], x[2], y[2], cosine_similarity(x[2], y[2]))
    print

这将显示以下输出：

('188.74.64.243', '99.229.98.18', '1', [0, 1, 1, 0], [0, 1, 1, 1], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '1', [0, 1, 1, 0], [1, 1, 1, 1], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '1', [0, 1, 1, 1], [1, 1, 1, 1], 0.8660254037844387)

('188.74.64.243', '99.229.98.18', '2', [0, 1, 1, 0], [0, 1, 1, 1], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '2', [0, 1, 1, 0], [1, 1, 1, 1], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '2', [0, 1, 1, 1], [1, 1, 1, 1], 0.8660254037844387)

如果列表未排序，即 ID 未分组在一起，则可以替换以下行：

for k, g in groupby(sorted(info_list, key=lambda x: x[1]), key=lambda x: x[1]):

Answer 2

保持你提供的数据（用字符串表示的向量），你可以编写一个函数，它接受你的两个元组，将字符串解压缩到一个 int 向量中，应用相似函数，然后重新打包.然后——将此函数与基本的嵌套循环一起使用：

import math

def cosine_similarity(v1,v2):
    sumxx, sumxy, sumyy = 0, 0, 0
    for i in range(len(v1)):
        x, y = v1[i],v2[i]
        sumxx += x*x
        sumyy += y*y
        sumxy += x*y
    return sumxy/math.sqrt(sumxx*sumyy)

def c_sim(t1,t2):
    ips1,id1,vlist1 = t1
    ips2,id2,vlist2 = t2
    v1 = [int(i) for i in vlist1[0].split(',')]
    v2 = [int(i) for i in vlist2[0].split(',')]
    if id1 == id2:
        return ips1,ips2,id1,vlist1,vlist2,cosine_similarity(v1,v2)

def process_list(data_list):
    n = len(data_list)
    ret_list = []
    for i in range(n-1):
        for j in range(i+1,n):
            t1,t2 = data_list[i],data_list[j]
            t = c_sim(t1,t2)
            if t: ret_list.append(t)
    return ret_list

data = [('188.74.64.243', '1', ['0, 1, 1, 0']),
('99.229.98.18',  '1', ['0, 1, 1, 1']),
('86.41.253.102', '1', ['1, 1, 1, 1']),
('188.74.64.243', '2', ['0, 1, 1, 0']),
('99.229.98.18',  '2', ['0, 1, 1, 1']),
('86.41.253.102', '2', ['1, 1, 1, 1'])]

for t in process_list(data): print(t)

输出：

('188.74.64.243', '99.229.98.18', '1', ['0, 1, 1, 0'], ['0, 1, 1, 1'], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '1', ['0, 1, 1, 0'], ['1, 1, 1, 1'], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '1', ['0, 1, 1, 1'], ['1, 1, 1, 1'], 0.8660254037844387)
('188.74.64.243', '99.229.98.18', '2', ['0, 1, 1, 0'], ['0, 1, 1, 1'], 0.8164965809277261)
('188.74.64.243', '86.41.253.102', '2', ['0, 1, 1, 0'], ['1, 1, 1, 1'], 0.7071067811865475)
('99.229.98.18', '86.41.253.102', '2', ['0, 1, 1, 1'], ['1, 1, 1, 1'], 0.8660254037844387)

Python 列表中所有可能对的余弦相似度

Python cosine-similarity on all possible pairs in list

python

cosine-similarity