使用 python 修复单词并删除拆分单词之间不需要的空格？

Question

我从文档中提取了句子列表。我正在预处理这个句子列表以使其更合理。我面临以下问题

我有Java is a prog rammng lan guage. C is a gen eral purpose la nguage.

这样的句子

我想用查字典来纠正这些句子？删除不需要的空格。

最终输出应该是Java is a programmng language. C is a general purpose language.

我需要一些帮助以寻找此类方法。如何解决上述问题？

我想用python代码解决上述问题。谢谢

Answer 1

如果要正确拼写和解析单词，则需要进行拼写检查。这是一个拼写检查器，通过导入 "re" 命名空间使用，以及完整的文章 here...

import re
from collections import Counter

def words(text): return re.findall(r'\w+', text.lower())

WORDS = Counter(words(open('big.txt').read()))

def P(word, N=sum(WORDS.values())): 
    "Probability of `word`."
    return WORDS[word] / N

def correction(word): 
    "Most probable spelling correction for `word`."
    return max(candidates(word), key=P)

def candidates(word): 
    "Generate possible spelling corrections for `word`."
    return (known([word]) or known(edits1(word)) or known(edits2(word)) or [word])

def known(words): 
    "The subset of `words` that appear in the dictionary of WORDS."
    return set(w for w in words if w in WORDS)

def edits1(word):
    "All edits that are one edit away from `word`."
    letters    = 'abcdefghijklmnopqrstuvwxyz'
    splits     = [(word[:i], word[i:])    for i in range(len(word) + 1)]
    deletes    = [L + R[1:]               for L, R in splits if R]
    transposes = [L + R[1] + R[0] + R[2:] for L, R in splits if len(R)>1]
    replaces   = [L + c + R[1:]           for L, R in splits if R for c in letters]
    inserts    = [L + c + R               for L, R in splits for c in letters]
    return set(deletes + transposes + replaces + inserts)

def edits2(word): 
    "All edits that are two edits away from `word`."
    return (e2 for e1 in edits1(word) for e2 in edits1(e1))`

不仅修复拆分词，还对"correct"不规则词进行删除、转置、插入等操作。您可以将 "big.txt" 文件替换为您在 Counter 构造函数中使用的文档，希望一切都能从那里开始工作。

Answer 2

这是一个适用于您的示例的简单脚本。显然，您需要更大的有效词语料库。此外，如果加入下一个词未能修复非词，您可能希望有一个 elif 分支回顾前一个词。

from string import punctuation

word_list = "big list of words including a programming language is general purpose"
valid_words = set(word_list.split())

bad = "Java is a prog ramming lan guage. C is a gen eral purpose la nguage."
words = bad.split()

out_words = []
i = 0
while i < len(words):
    word = words[i]
    if word not in valid_words and i+1 < len(words):
        next_word = words[i+1]
        joined = word + next_word
        if joined.strip(punctuation) in valid_words:
            word = joined
            i += 1
    out_words.append(word)
    i += 1

good = " ".join(out_words)
print(good)

使用 python 修复单词并删除拆分单词之间不需要的空格？

Fix the words and remove the unwanted spaces between splitted word using python?

python

python-3.x

split

space