Spacy：如何在特殊情况标记化规则列表中添加冒号字符

Question

我有下面这句话：

'25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?'

我想将冒号与其余单词分开。

默认情况下，这里是 Spacy returns:

print([w.text for w in nlp('25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?')])

['25', ')', 'Figure', '9:“lines', 'are', 'results', 'of', 'two', '-', 'step', 'adsorption', 'model', '”', '-', '>', 'What', 'method', '/', 'software', 'was', 'used', 'for', 'the', 'curve', 'fitting', '?']

我想得到的是：

['25', ')', 'Figure', '9', ':', '“', lines', 'are', 'results', 'of', 'two', '-', 'step', 'adsorption', 'model', '”', '-', '>', 'What', 'method', '/', 'software', 'was', 'used', 'for', 'the', 'curve', 'fitting', '?']

我正在尝试做：

# Add special case rule
special_case = [{ORTH: ":"}]
nlp.tokenizer.add_special_case(":", special_case)

但没有结果，打印显示相同的值。

Answer 1

只需使用word_tokenize

from nltk.tokenize import word_tokenize 
string_my='25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?'
word_tokenize(string_my) 

['25', ')', 'Figure', '9', ':', '“', 'lines', 'are', 'results', 'of', 'two-step', 'adsorption', 'model', '”', '-', '>', 'What', 'method/software', 'was', 'used', 'for', 'the', 'curve', 'fitting', '?']

Answer 2

尝试将 nlp.tokenizer.infix_finditer 修改为 compile_infix_regex:

import spacy
from spacy.util import compile_infix_regex

text = "'25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?'"
 
nlp = spacy.load("en_core_web_md")
infixes = (":",) + nlp.Defaults.infixes
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

doc = nlp(text)

for tok in doc:
    print(tok, end =", ")

', 25, ), Figure, 9, :, “lines, are, results, of, two, -, step, adsorption, model, ”, -, >, What, method, /, software, was, used, for, the, curve, fitting, ?, ',

Spacy：如何在特殊情况标记化规则列表中添加冒号字符

Spacy: how to add the colon character in the list of special case tokenization rules

python

stringtokenizer

spacy