替换 python 中的 apostrophe/short 个单词
Replace apostrophe/short words in python
我正在使用 python 来清理给定的句子。假设我的句子是：
What's the best way to ensure this?
我要转换：
What's -> What is
同样，
must've -> must have
还有，动词原形，
told -> tell
单数变复数，依此类推
我目前正在探索 textblob。但并非以上所有都可以使用它。
如果你想自己滚动，你可以用它来做收缩映射：
http://alicebot.blogspot.com/2009/03/english-contractions-and-expansions.html
这用于动词替换：
http://www.lexically.net/downloads/BNC_wordlists/e_lemma.txt
对于后者，您可能希望生成一个反向字典，将所有共轭形式映射到它们的原始形式（可能要记住可能存在模棱两可的形式，因此请确保检查这些形式并正确处理它们） .
对于第一个问题，没有直接的模块可以为您完成，所以您必须自己构建，首先您需要像这样的收缩字典：
contractions = {
"ain't": "am not / are not",
"aren't": "are not / am not",
"can't": "cannot",
"can't've": "cannot have",
"'cause": "because",
"could've": "could have",
"couldn't": "could not",
"couldn't've": "could not have",
"didn't": "did not",
"doesn't": "does not",
"don't": "do not",
"hadn't": "had not",
"hadn't've": "had not have",
"hasn't": "has not",
"haven't": "have not",
"he'd": "he had / he would",
"he'd've": "he would have",
"he'll": "he shall / he will",
"he'll've": "he shall have / he will have",
"he's": "he has / he is",
"how'd": "how did",
"how'd'y": "how do you",
"how'll": "how will",
"how's": "how has / how is",
"i'd": "I had / I would",
"i'd've": "I would have",
"i'll": "I shall / I will",
"i'll've": "I shall have / I will have",
"i'm": "I am",
"i've": "I have",
"isn't": "is not",
"it'd": "it had / it would",
"it'd've": "it would have",
"it'll": "it shall / it will",
"it'll've": "it shall have / it will have",
"it's": "it has / it is",
"let's": "let us",
"ma'am": "madam",
"mayn't": "may not",
"might've": "might have",
"mightn't": "might not",
"mightn't've": "might not have",
"must've": "must have",
"mustn't": "must not",
"mustn't've": "must not have",
"needn't": "need not",
"needn't've": "need not have",
"o'clock": "of the clock",
"oughtn't": "ought not",
"oughtn't've": "ought not have",
"shan't": "shall not",
"sha'n't": "shall not",
"shan't've": "shall not have",
"she'd": "she had / she would",
"she'd've": "she would have",
"she'll": "she shall / she will",
"she'll've": "she shall have / she will have",
"she's": "she has / she is",
"should've": "should have",
"shouldn't": "should not",
"shouldn't've": "should not have",
"so've": "so have",
"so's": "so as / so is",
"that'd": "that would / that had",
"that'd've": "that would have",
"that's": "that has / that is",
"there'd": "there had / there would",
"there'd've": "there would have",
"there's": "there has / there is",
"they'd": "they had / they would",
"they'd've": "they would have",
"they'll": "they shall / they will",
"they'll've": "they shall have / they will have",
"they're": "they are",
"they've": "they have",
"to've": "to have",
"wasn't": "was not",
"we'd": "we had / we would",
"we'd've": "we would have",
"we'll": "we will",
"we'll've": "we will have",
"we're": "we are",
"we've": "we have",
"weren't": "were not",
"what'll": "what shall / what will",
"what'll've": "what shall have / what will have",
"what're": "what are",
"what's": "what has / what is",
"what've": "what have",
"when's": "when has / when is",
"when've": "when have",
"where'd": "where did",
"where's": "where has / where is",
"where've": "where have",
"who'll": "who shall / who will",
"who'll've": "who shall have / who will have",
"who's": "who has / who is",
"who've": "who have",
"why's": "why has / why is",
"why've": "why have",
"will've": "will have",
"won't": "will not",
"won't've": "will not have",
"would've": "would have",
"wouldn't": "would not",
"wouldn't've": "would not have",
"y'all": "you all",
"y'all'd": "you all would",
"y'all'd've": "you all would have",
"y'all're": "you all are",
"y'all've": "you all have",
"you'd": "you had / you would",
"you'd've": "you would have",
"you'll": "you shall / you will",
"you'll've": "you shall have / you will have",
"you're": "you are",
"you've": "you have"
}
然后写一些代码根据字典修改你的文本，像这样：
text="What's the best way to ensure this?"
for word in text.split():
if word.lower() in contractions:
text = text.replace(word, contractions[word.lower()])
print(text)
关于改变动词时态的第二个问题，nodebox's linguistics library is very popular and highly recommended for such tasks. After downloading their zip file，解压缩并将其复制到 python 的站点包目录。完成之后，你可以这样写：
import en
for word in text.split():
if en.is_verb(word.lower()):
text = text.replace(word, en.verb.present(word.lower()))
print text
注意：此库仅适用于 Python 2，因为它尚未提供对 python 3 的支持。
上面的答案将非常有效，并且对于模棱两可的收缩可能会更好（尽管我认为没有那么多模棱两可的情况）。我会使用更具可读性和更易于维护的东西：
import re
def decontracted(phrase):
# specific
phrase = re.sub(r"won\'t", "will not", phrase)
phrase = re.sub(r"can\'t", "can not", phrase)
# general
phrase = re.sub(r"n\'t", " not", phrase)
phrase = re.sub(r"\'re", " are", phrase)
phrase = re.sub(r"\'s", " is", phrase)
phrase = re.sub(r"\'d", " would", phrase)
phrase = re.sub(r"\'ll", " will", phrase)
phrase = re.sub(r"\'t", " not", phrase)
phrase = re.sub(r"\'ve", " have", phrase)
phrase = re.sub(r"\'m", " am", phrase)
return phrase
test = "Hey I'm Yann, how're you and how's it going ? That's interesting: I'd love to hear more about it."
print(decontracted(test))
# Hey I am Yann, how are you and how is it going ? That is interesting: I would love to hear more about it.
它可能有一些我没有想到的缺陷。
这可能不适合您的特定解决方案，但（对于一般知识）有一个很棒的 开源软件库，名为 Spacy。在类似情况下，它使生活更轻松。演示：
texts = ["what's", "must've", "told"]
for text in texts:
doc = nlp(text)
lemmatized_text = ' '.join([token.lemma_ for token in doc])
print(lemmatized_text)
输出：
what be
must have
tell
我正在使用 python 来清理给定的句子。假设我的句子是：
What's the best way to ensure this?
我要转换：
What's -> What is
同样，
must've -> must have
还有，动词原形，
told -> tell
单数变复数，依此类推
我目前正在探索 textblob。但并非以上所有都可以使用它。
如果你想自己滚动，你可以用它来做收缩映射：
http://alicebot.blogspot.com/2009/03/english-contractions-and-expansions.html
这用于动词替换：
http://www.lexically.net/downloads/BNC_wordlists/e_lemma.txt
对于后者，您可能希望生成一个反向字典，将所有共轭形式映射到它们的原始形式（可能要记住可能存在模棱两可的形式，因此请确保检查这些形式并正确处理它们） .
对于第一个问题，没有直接的模块可以为您完成，所以您必须自己构建，首先您需要像这样的收缩字典：
contractions = {
"ain't": "am not / are not",
"aren't": "are not / am not",
"can't": "cannot",
"can't've": "cannot have",
"'cause": "because",
"could've": "could have",
"couldn't": "could not",
"couldn't've": "could not have",
"didn't": "did not",
"doesn't": "does not",
"don't": "do not",
"hadn't": "had not",
"hadn't've": "had not have",
"hasn't": "has not",
"haven't": "have not",
"he'd": "he had / he would",
"he'd've": "he would have",
"he'll": "he shall / he will",
"he'll've": "he shall have / he will have",
"he's": "he has / he is",
"how'd": "how did",
"how'd'y": "how do you",
"how'll": "how will",
"how's": "how has / how is",
"i'd": "I had / I would",
"i'd've": "I would have",
"i'll": "I shall / I will",
"i'll've": "I shall have / I will have",
"i'm": "I am",
"i've": "I have",
"isn't": "is not",
"it'd": "it had / it would",
"it'd've": "it would have",
"it'll": "it shall / it will",
"it'll've": "it shall have / it will have",
"it's": "it has / it is",
"let's": "let us",
"ma'am": "madam",
"mayn't": "may not",
"might've": "might have",
"mightn't": "might not",
"mightn't've": "might not have",
"must've": "must have",
"mustn't": "must not",
"mustn't've": "must not have",
"needn't": "need not",
"needn't've": "need not have",
"o'clock": "of the clock",
"oughtn't": "ought not",
"oughtn't've": "ought not have",
"shan't": "shall not",
"sha'n't": "shall not",
"shan't've": "shall not have",
"she'd": "she had / she would",
"she'd've": "she would have",
"she'll": "she shall / she will",
"she'll've": "she shall have / she will have",
"she's": "she has / she is",
"should've": "should have",
"shouldn't": "should not",
"shouldn't've": "should not have",
"so've": "so have",
"so's": "so as / so is",
"that'd": "that would / that had",
"that'd've": "that would have",
"that's": "that has / that is",
"there'd": "there had / there would",
"there'd've": "there would have",
"there's": "there has / there is",
"they'd": "they had / they would",
"they'd've": "they would have",
"they'll": "they shall / they will",
"they'll've": "they shall have / they will have",
"they're": "they are",
"they've": "they have",
"to've": "to have",
"wasn't": "was not",
"we'd": "we had / we would",
"we'd've": "we would have",
"we'll": "we will",
"we'll've": "we will have",
"we're": "we are",
"we've": "we have",
"weren't": "were not",
"what'll": "what shall / what will",
"what'll've": "what shall have / what will have",
"what're": "what are",
"what's": "what has / what is",
"what've": "what have",
"when's": "when has / when is",
"when've": "when have",
"where'd": "where did",
"where's": "where has / where is",
"where've": "where have",
"who'll": "who shall / who will",
"who'll've": "who shall have / who will have",
"who's": "who has / who is",
"who've": "who have",
"why's": "why has / why is",
"why've": "why have",
"will've": "will have",
"won't": "will not",
"won't've": "will not have",
"would've": "would have",
"wouldn't": "would not",
"wouldn't've": "would not have",
"y'all": "you all",
"y'all'd": "you all would",
"y'all'd've": "you all would have",
"y'all're": "you all are",
"y'all've": "you all have",
"you'd": "you had / you would",
"you'd've": "you would have",
"you'll": "you shall / you will",
"you'll've": "you shall have / you will have",
"you're": "you are",
"you've": "you have"
}
然后写一些代码根据字典修改你的文本，像这样：
text="What's the best way to ensure this?"
for word in text.split():
if word.lower() in contractions:
text = text.replace(word, contractions[word.lower()])
print(text)
关于改变动词时态的第二个问题，nodebox's linguistics library is very popular and highly recommended for such tasks. After downloading their zip file，解压缩并将其复制到 python 的站点包目录。完成之后，你可以这样写：
import en
for word in text.split():
if en.is_verb(word.lower()):
text = text.replace(word, en.verb.present(word.lower()))
print text
注意：此库仅适用于 Python 2，因为它尚未提供对 python 3 的支持。
上面的答案将非常有效，并且对于模棱两可的收缩可能会更好（尽管我认为没有那么多模棱两可的情况）。我会使用更具可读性和更易于维护的东西：
import re
def decontracted(phrase):
# specific
phrase = re.sub(r"won\'t", "will not", phrase)
phrase = re.sub(r"can\'t", "can not", phrase)
# general
phrase = re.sub(r"n\'t", " not", phrase)
phrase = re.sub(r"\'re", " are", phrase)
phrase = re.sub(r"\'s", " is", phrase)
phrase = re.sub(r"\'d", " would", phrase)
phrase = re.sub(r"\'ll", " will", phrase)
phrase = re.sub(r"\'t", " not", phrase)
phrase = re.sub(r"\'ve", " have", phrase)
phrase = re.sub(r"\'m", " am", phrase)
return phrase
test = "Hey I'm Yann, how're you and how's it going ? That's interesting: I'd love to hear more about it."
print(decontracted(test))
# Hey I am Yann, how are you and how is it going ? That is interesting: I would love to hear more about it.
它可能有一些我没有想到的缺陷。
这可能不适合您的特定解决方案，但（对于一般知识）有一个很棒的 开源软件库，名为 Spacy。在类似情况下，它使生活更轻松。演示：
texts = ["what's", "must've", "told"]
for text in texts:
doc = nlp(text)
lemmatized_text = ' '.join([token.lemma_ for token in doc])
print(lemmatized_text)
输出：
what be
must have
tell