如何避免使用 Matcher 在 SpaCy 中重复提取重叠模式？

Question

我需要通过 python Spacy Matcher 从 2 个列表中提取项目组合。问题如下：让我们有 2 个列表：

colors=['red','bright red','black','brown','dark brown']
animals=['fox','bear','hare','squirrel','wolf']

我通过以下代码匹配序列：

first_color=[]
last_color=[]
only_first_color=[]
for color in colors:
    if ' ' in color:
        first_color.append(color.split(' ')[0])
        last_color.append(color.split(' ')[1])
    else:
        only_first_color.append(color)
matcher = Matcher(nlp.vocab)

pattern1 = [{"TEXT": {"IN": only_first_color}},{"TEXT":{"IN": animals}}]
pattern2 = [{"TEXT": {"IN": first_color}},{"TEXT": {"IN": last_color}},{"TEXT":{"IN": animals}}]

matcher.add("ANIMALS", None, pattern1,pattern2)

doc = nlp('bright red fox met black wolf')

matches = matcher(doc)

for match_id, start, end in matches:
    string_id = nlp.vocab.strings[match_id]  # Get string representation
    span = doc[start:end]  # The matched span
    print(start, end, span.text)

它给出了输出：

0 3 bright red fox
1 3 red fox
4 6 black wolf

如何只提取 'bright red fox' 和 'black wolf'？我应该更改模式规则还是 post-处理匹配项？

任何想法表示赞赏！

Answer 1

您可以使用 spacy.util.filter_spans:

Filter a sequence of Span objects and remove duplicates or overlaps. Useful for creating named entities (where one token can only be part of one entity) or when merging spans with Retokenizer.merge. When spans overlap, the (first) longest span is preferred over shorter spans.

Python代码：

matches = matcher(doc)
spans = [doc[start:end] for _, start, end in matches]
for span in spacy.util.filter_spans(spans):
    print(span.start, span.end, span.text)

输出：

0 3 bright red fox
4 6 black wolf

如何避免使用 Matcher 在 SpaCy 中重复提取重叠模式？

How to avoid double-extracting of overlapping patterns in SpaCy with Matcher?

python

nlp

spacy

matcher