为什么 NLTK Lemmatizer 不能对一些复数词进行词形还原?
Why NLTK Lemmatizer can't lemmatize some plural words?
我试图对《古兰经》中的一个词进行词形还原,但有些词无法进行词形还原。
这是我的句子:
sentence = "Then bring ten surahs like it that have been invented and call upon for assistance whomever you can besides Allah if you should be truthful"
那句话是我的 txt 数据集的一部分。
如您所见,"surahs" 是 "surah" 的复数形式。
我试过我的代码:
def lemmatize(self, ayat):
wordnet_lemmatizer = WordNetLemmatizer()
result = []
for i in xrange (len(ayat)):
result.append(wordnet_lemmatizer.lemmatize(sentence[i],'v'))
return result
当我运行打印时,结果是这样的:
['bring', 'ten', 'surahs', 'like', u'invent', 'call', 'upon', 'assistance', 'whomever', 'besides', 'Allah', 'truthful']
'surahs'没有变成'surah'。
谁能告诉我为什么?谢谢。
见
- Stemming some plurals with wordnet lemmatizer doesn't work
- Python NLTK Lemmatization of the word 'further' with wordnet
对于大多数非标准英语单词,WordNet Lemmatizer 对获得正确的词条帮助不大,请尝试词干分析器:
>>> from nltk.stem import PorterStemmer
>>> porter = PorterStemmer()
>>> porter.stem('surahs')
u'surah'
此外,尝试 earthy
中的 lemmatize_sent
(nltk
包装器,"shameless plug"):
>>> from earthy.nltk_wrappers import lemmatize_sent
>>> sentence = "Then bring ten surahs like it that have been invented and call upon for assistance whomever you can besides Allah if you should be truthful"
>>> lemmatize_sent(sentence)
[('Then', 'Then', 'RB'), ('bring', 'bring', 'VBG'), ('ten', 'ten', 'RP'), ('surahs', 'surahs', 'NNS'), ('like', 'like', 'IN'), ('it', 'it', 'PRP'), ('that', 'that', 'WDT'), ('have', 'have', 'VBP'), ('been', u'be', 'VBN'), ('invented', u'invent', 'VBN'), ('and', 'and', 'CC'), ('call', 'call', 'VB'), ('upon', 'upon', 'NN'), ('for', 'for', 'IN'), ('assistance', 'assistance', 'NN'), ('whomever', 'whomever', 'NN'), ('you', 'you', 'PRP'), ('can', 'can', 'MD'), ('besides', 'besides', 'VB'), ('Allah', 'Allah', 'NNP'), ('if', 'if', 'IN'), ('you', 'you', 'PRP'), ('should', 'should', 'MD'), ('be', 'be', 'VB'), ('truthful', 'truthful', 'JJ')]
>>> words, lemmas, tags = zip(*lemmatize_sent(sentence))
>>> lemmas
('Then', 'bring', 'ten', 'surahs', 'like', 'it', 'that', 'have', u'be', u'invent', 'and', 'call', 'upon', 'for', 'assistance', 'whomever', 'you', 'can', 'besides', 'Allah', 'if', 'you', 'should', 'be', 'truthful')
>>> from earthy.nltk_wrappers import pywsd_lemmatize
>>> pywsd_lemmatize('surahs')
'surahs'
>>> from earthy.nltk_wrappers import porter_stem
>>> porter_stem('surahs')
u'surah'
我试图对《古兰经》中的一个词进行词形还原,但有些词无法进行词形还原。
这是我的句子:
sentence = "Then bring ten surahs like it that have been invented and call upon for assistance whomever you can besides Allah if you should be truthful"
那句话是我的 txt 数据集的一部分。 如您所见,"surahs" 是 "surah" 的复数形式。 我试过我的代码:
def lemmatize(self, ayat):
wordnet_lemmatizer = WordNetLemmatizer()
result = []
for i in xrange (len(ayat)):
result.append(wordnet_lemmatizer.lemmatize(sentence[i],'v'))
return result
当我运行打印时,结果是这样的:
['bring', 'ten', 'surahs', 'like', u'invent', 'call', 'upon', 'assistance', 'whomever', 'besides', 'Allah', 'truthful']
'surahs'没有变成'surah'。
谁能告诉我为什么?谢谢。
见
- Stemming some plurals with wordnet lemmatizer doesn't work
- Python NLTK Lemmatization of the word 'further' with wordnet
对于大多数非标准英语单词,WordNet Lemmatizer 对获得正确的词条帮助不大,请尝试词干分析器:
>>> from nltk.stem import PorterStemmer
>>> porter = PorterStemmer()
>>> porter.stem('surahs')
u'surah'
此外,尝试 earthy
中的 lemmatize_sent
(nltk
包装器,"shameless plug"):
>>> from earthy.nltk_wrappers import lemmatize_sent
>>> sentence = "Then bring ten surahs like it that have been invented and call upon for assistance whomever you can besides Allah if you should be truthful"
>>> lemmatize_sent(sentence)
[('Then', 'Then', 'RB'), ('bring', 'bring', 'VBG'), ('ten', 'ten', 'RP'), ('surahs', 'surahs', 'NNS'), ('like', 'like', 'IN'), ('it', 'it', 'PRP'), ('that', 'that', 'WDT'), ('have', 'have', 'VBP'), ('been', u'be', 'VBN'), ('invented', u'invent', 'VBN'), ('and', 'and', 'CC'), ('call', 'call', 'VB'), ('upon', 'upon', 'NN'), ('for', 'for', 'IN'), ('assistance', 'assistance', 'NN'), ('whomever', 'whomever', 'NN'), ('you', 'you', 'PRP'), ('can', 'can', 'MD'), ('besides', 'besides', 'VB'), ('Allah', 'Allah', 'NNP'), ('if', 'if', 'IN'), ('you', 'you', 'PRP'), ('should', 'should', 'MD'), ('be', 'be', 'VB'), ('truthful', 'truthful', 'JJ')]
>>> words, lemmas, tags = zip(*lemmatize_sent(sentence))
>>> lemmas
('Then', 'bring', 'ten', 'surahs', 'like', 'it', 'that', 'have', u'be', u'invent', 'and', 'call', 'upon', 'for', 'assistance', 'whomever', 'you', 'can', 'besides', 'Allah', 'if', 'you', 'should', 'be', 'truthful')
>>> from earthy.nltk_wrappers import pywsd_lemmatize
>>> pywsd_lemmatize('surahs')
'surahs'
>>> from earthy.nltk_wrappers import porter_stem
>>> porter_stem('surahs')
u'surah'