如何构建一个简单的分词器

Question

我想知道如何构建一个非常简单的分词器。给定字典 d（在本例中为列表）和句子 s，我想 return 句子的所有可能标记（=单词）。这是我尝试过的：

l = ["the","snow","ball","snowball","is","cold"]
sentence = "thesnowballisverycold"

def subs(string, ret=['']):
    if len(string) == 0:
        return ret
    head, tail = string[0], string[1:]
    ret = ret + list(map(lambda x: x+head, ret))
    return subs(tail, ret)
    
print((list(set(subs(sentence))&set(l))))

但是这个 returns:

["snow","ball","cold","is","snowball","the"]

我可以比较子串，但一定有更好的方法，对吧？我想要的：

["the","snowball","is","cold"]

Answer 1

您可以在此处使用正则表达式：

import re
l = ["the","snow","ball","snowball","is","cold"]
pattern = "|".join(sorted(l, key=len, reverse=True))
sentence = "thesnowballisverycold"
print( re.findall(pattern, sentence) )
# => ['the', 'snowball', 'is', 'cold']

参见Python demo。

如何构建一个简单的分词器

How to build a simple tokenizer

python

substring

tokenize