如何删除非单词边界的停用词？

Question

我正在尝试使用我的停用词列表删除 txt 文件中的停用词。有些停用词已被删除，有些则没有。

例如这句话："it taste nice, doesn’t it?" 应该有类似 "taste nice" 的输出，但我的代码输出："taste nice doesnt it "

我的停用词列表来自：https://www.ranks.nl/stopwords（长停用词列表）。

这是我的代码：

    public static void main(String[] args) {

    ArrayList sw = new ArrayList<>();

    try{
        FileInputStream fis = new FileInputStream("/Users/Dan/Desktop/DATA/stopwords.txt");

        byte b[] = new byte[fis.available()];
        fis.read(b);
            fis.close();

            String data[] = new String(b).split("\n");

        for(int i = 0; i < data.length; i++)
        {
            sw.add(data[i].trim());
        }
         FileInputStream fis2 = new FileInputStream("/Users/Dan/Desktop/DATA/cleandata.txt");

        byte bb[] = new byte[fis2.available()];
        fis2.read(bb);
            fis2.close();

            String data2[] = new String(bb).split("\n");



            for(int i = 0; i < data2.length; i++)

        {
            String file = "";
            String s[] = data2[i].split("\s");
            for(int j = 0; j < s.length; j++)
            {
                if(!(sw.contains(s[j].trim().toLowerCase())))
                {
                    file=file + s[j] + " ";
                }

            }
            file = file.replaceAll("[^a-zA-Z\s+]", "");

            System.out.println(file.replaceAll("\s+", " ").toLowerCase() + "\n");

        }

    } catch(Exception a){
        a.printStackTrace();
    }

}

我该怎么办？我觉得打印有问题

file = file.replaceAll("[^a-zA-Z\s+]", "");

System.out.println(file.replaceAll("\s+", " ").toLowerCase() + "\n");

Answer 1

使用了两个不同的引号字符。停用词文件包含 doesn't，您的输入包含 doesn’t。

因为引号不一样，所以不匹配

编辑：这是一个稍微重构的解决方案，它生成正确的输出（如果你不在输入中使用奇怪的引号）。

import java.util.Arrays;
import java.util.HashSet;
import java.util.Scanner;
import java.util.Set;
import java.util.stream.Collectors;

public class StopWordsApp {

    // the platform-specific end of line token
    private static final String EOL = String.format("%n");

    private final Set<String> stopWords = new HashSet<>(Arrays.asList(readLines("stopwords.txt")));

    public static void main(String[] args) {
        StopWordsApp stopWordsApp = new StopWordsApp();
        String[] lines = readLines("cleandata.txt");
        printLines(stopWordsApp.removeStopWords(lines));
    }

    private String[] removeStopWords(String[] inputLines) {
        return Arrays.stream(inputLines)
                // map the String array to a Line object
                .map(Line::new)
                // map the Line to a String without stop words
                .map(this::removeStopWords)
                // convert the stream to an array
                .toArray(String[]::new);
    }

    private String removeStopWords(Line line) {
        return line.words().stream()
                // map the word to its normalized version
                .map(Word::normalized)
                // remove stop words
                .filter(n -> !stopWords.contains(n))
                // join into a String separated by spaces
                .collect(Collectors.joining(" "));
    }

    private static String[] readLines(String fileName) {
        return readFile(fileName).split(EOL);
    }

    private static String readFile(String fileName) {
        return new Scanner(StopWordsApp.class.getResourceAsStream(fileName), "UTF-8").useDelimiter("\A").next();
    }

    private static void printLines(String[] lines) {
        for (String line : lines) {
            System.out.println(line);
        }
    }
}

我为一行提取了单独的 classes：

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class Line {

    private final List<Word> words;

    public Line(String input) {
        String[] wordInputs = input.split("\s+");
        words = Arrays.stream(wordInputs)
                // remove empty Strings
                .filter(v -> !v.isEmpty())
                // map String to a Word object
                .map(Word::new)
                // collect into a List
                .collect(Collectors.toList());
    }

    public List<Word> words() {
        return words;
    }

}

..一句话：

public class Word {

    private final String normalized;

    public Word(String input) {
        normalized = input
                // convert to lower case
                .toLowerCase()
                // remove everything that's not a lower case letter or a quote
                // (the stopwords file only contains lower case letters and quotes)
                .replaceAll("[^a-z']", "")
                // replace consecutive white space with a single space
                .replaceAll("\s+", " ")
                // trim any white space at the edges
                .trim();
    }

    public String normalized() {
        return normalized;
    }

}

...和自定义（运行时）异常：

public class StopWordsException extends RuntimeException {
    public StopWordsException(Exception e) {
        super(e);
    }
}

我到处都使用了 Java 8 个流，并添加了评论来解释发生了什么。

随着输入：

it Taste nice, doesn't it?

输出为：

taste nice

P.S。文件 'stopwords.txt' 和 'cleandata.txt' 需要与 StopWordsApp class.

在同一个包中

如何删除非单词边界的停用词？

How can I remove the stop words with a non-word boundary?

java

stop-words