如何使用 Spacy 按句子分解文档

How to break up document by sentences with Spacy

如何将文档(例如段落、书籍等)分解成句子。

例如,"The dog ran. The cat jumped" 变成 ["The dog ran", "The cat jumped"] with spacy?

来自 spacy's github support page

from __future__ import unicode_literals, print_function
from spacy.en import English

raw_text = 'Hello, world. Here are two sentences.'
nlp = English()
doc = nlp(raw_text)
sentences = [sent.string.strip() for sent in doc.sents]

最新的答案是这样的:

from __future__ import unicode_literals, print_function
from spacy.lang.en import English # updated

raw_text = 'Hello, world. Here are two sentences.'
nlp = English()
nlp.add_pipe(nlp.create_pipe('sentencizer')) # updated
doc = nlp(raw_text)
sentences = [sent.string.strip() for sent in doc.sents]

回答

import spacy
nlp = spacy.load('en_core_web_sm')

text = 'My first birthday was great. My 2. was even better.'
sentences = [i for i in nlp(text).sents]

附加信息
这假定您已经在系统上安装了模型 "en_core_web_sm"。如果没有,您可以在终端中通过 运行 以下命令轻松安装它:

$ python -m spacy download en_core_web_sm

(有关所有可用模型的概述,请参阅 here。)

根据您的数据,这可能会比仅使用 spacy.lang.en.English 产生更好的结果。一个(非常简单的)比较示例:

import spacy
from spacy.lang.en import English

nlp_simple = English()
nlp_simple.add_pipe(nlp_simple.create_pipe('sentencizer'))

nlp_better = spacy.load('en_core_web_sm')


text = 'My first birthday was great. My 2. was even better.'

for nlp in [nlp_simple, nlp_better]:
    for i in nlp(text).sents:
        print(i)
    print('-' * 20)

输出:

>>> My first birthday was great.
>>> My 2.
>>> was even better.
>>> --------------------
>>> My first birthday was great.
>>> My 2. was even better.
>>> --------------------

对于 spacy 3.0.1,他们改变了管道。

from spacy.lang.en import English 

nlp = English()
nlp.add_pipe('sentencizer')


def split_in_sentences(text):
    doc = nlp(text)
    return [str(sent).strip() for sent in doc.sents]

已更新以反映第一个答案中的评论

from spacy.lang.en import English

raw_text = 'Hello, world. Here are two sentences.'
nlp = English()
nlp.add_pipe('sentencizer')
doc = nlp(raw_text)
sentences = [sent.text.strip() for sent in doc.sents]