在余弦相似度中对数字应用比字符串更多的权重

Question

我有一个程序可以从互联网上提取地址并根据数据库检查它们。它很有用，但我现在正尝试引入一个相似性函数来比较 Internet 上的地址和我数据库中的地址。

我正在使用以下脚本来检查余弦相似度与地址的比较情况：

import string
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import CountVectorizer

addresses = [
  '705 Sherlock House, 221B Baker Street, London NW1 6XE', 
  '75 Sherlock House, 221B Baker Street, London NW1 6XE', 
  'Apartment 704 Sherlock House, 221B Baker Street, London NW1 6XE', 
  'Apartment 705 Sherlock House, 221B Baker Street, London NW1 6XE', 
  '705, 221B Baker Street, London NW1 6XE', 
  '75, 221B Baker Street, London NW1 6XE',
  '705 Watson House, 219 Baker Street, London NW1 6XE',
  '32 Baker Street, London NW1 6XE',
  '1060 West Addison, London, W2 6SR',
  '705 Sherlock Hse, Baker Street, London, NW1'
  ]

def clean_address(text):
  text = ''.join([word for word in text if word not in string.punctuation])
  text = text.lower()
  return text

cleaned = list(map(clean_address, addresses))

vectorizer = CountVectorizer()
transformedVectorizer = vectorizer.fit_transform(cleaned)
vectors = transformedVectorizer.toarray()

csim = cosine_similarity(vectors)

def cosine_sim_vectors(vec1, vec2):
  vec1 = vec1.reshape(1, -1)
  vec2 = vec2.reshape(1, -1)

  return cosine_similarity(vec1, vec2)[0][0]

cosine_sim_vectors1 = cosine_sim_vectors(vectors[0], vectors[1])
cosine_sim_vectors2 = cosine_sim_vectors(vectors[0], vectors[2])
cosine_sim_vectors3 = cosine_sim_vectors(vectors[0], vectors[3])
cosine_sim_vectors4 = cosine_sim_vectors(vectors[0], vectors[4])
cosine_sim_vectors5 = cosine_sim_vectors(vectors[0], vectors[5])
cosine_sim_vectors6 = cosine_sim_vectors(vectors[0], vectors[6])
cosine_sim_vectors7 = cosine_sim_vectors(vectors[0], vectors[7])
cosine_sim_vectors8 = cosine_sim_vectors(vectors[0], vectors[8])
cosine_sim_vectors9 = cosine_sim_vectors(vectors[0], vectors[9])

print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 75 Sherlock House, 221B Baker Street, London NW1 6XE".format(cosine_sim_vectors1 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to Apartment 704 Sherlock House, 221B Baker Street, London NW1 6XE".format(cosine_sim_vectors2 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to Apartment 705 Sherlock House, 221B Baker Street, London NW1 6XE".format(cosine_sim_vectors3 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 705, 221B Baker Street, London NW1 6XE".format(cosine_sim_vectors4 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 75, 221B Baker Street, London NW1 6XE".format(cosine_sim_vectors5 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 705 Watson House, 219 Baker Street, London NW1 6XE".format(cosine_sim_vectors6 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 32 Baker Street, London NW1 6XE".format(cosine_sim_vectors7 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 1060 West Addison, London, W2 6SR".format(cosine_sim_vectors8 * 100))
print("705 Sherlock House, 221B Baker Street, London NW1 6XE is {:.1f}% similar to 705 Sherlock Hse, Baker Street, London, NW1".format(cosine_sim_vectors9 * 100))

输出为：

705 Sherlock House, 221B Baker Street, London NW1 6XE is 88.9% similar to 75 Sherlock House, 221B Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 84.3% similar to Apartment 704 Sherlock House, 221B Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 94.9% similar to Apartment 705 Sherlock House, 221B Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 88.2% similar to 705, 221B Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 75.6% similar to 75, 221B Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 77.8% similar to 705 Watson House, 219 Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 68.0% similar to 32 Baker Street, London NW1 6XE
705 Sherlock House, 221B Baker Street, London NW1 6XE is 13.6% similar to 1060 West Addison, London, W2 6SR
705 Sherlock House, 221B Baker Street, London NW1 6XE is 75.6% similar to 705 Sherlock Hse, Baker Street, London, NW1

它做得很合理，因为我可能会关注超过 60-70% 的任何东西，我印象深刻，它几乎抓住了我用 705 Watson House 和 705 Sherlock Hse 故意欺骗它的企图，但我确实认为它会改进算法，例如，如果它认识到 705 比伦敦更重要，或者考虑到我可以删除伦敦，6XE。

如果有更合适的相似度函数，我也愿意使用其他相似度函数，因为我知道余弦相似度正在将字符串更改为向量，并且基本上平等对待它们。

Answer 1

我的地址字符串的一部分比另一部分增加更多的权重没有任何好处，余弦相似性可以开箱即用。

为此，余弦相似度是比字符串编辑距离更好的算法，因为“75 Sherlock House, 221B Baker Street, London NW1 6XE”与“705 Sherlock House, 221B Baker Street, London NW1 6XE”并不更相似比 'Apartment 705 Sherlock House, 221B Baker Street, London NW1 6XE' - 余弦相似度符合这种直觉。

在余弦相似度中对数字应用比字符串更多的权重

Applying more weight to numbers than strings in Cosine Similarity

python

cosine-similarity

scikit-learn