哈希映射，删除重复项，但也存储原始事件的数量。 java

Question

我有一个 java 哈希映射，用于生成通过推理学习的 "rules"。例如，输入可能如下所示：

'prevents'('scurvy','vitamin C').
'contains'('vitamin C','orange').
'contains'('vitamin C','sauerkraut').
'isa'('fruit','orange').
'improves'('health','fruit').

输出可能如下所示：

prevents(scurvy, orange).
prevents(scurvy, sauerkraut).
improves(health, orange).

对于小型测试，一切都运行良好，但在我的实际数据集中，我有许多相同规则的实例。我想以某种方式存储每个规则的出现次数并将其写入文件，以及它被看到的次数，因为我认为这可能是某种天真的置信度衡量规则的可能性.

此时我存储的句子是这样的：

public class Sentence {
private String verb;
private String object;
private String subject;
public Sentence(String verb, String object, String subject ){
this.verb = verb;
this.object = object;
this.subject = subject;
}
public String getVerb(){ return verb; }
public String getObject(){ return object; }
public String getSubject(){ return subject; }
public String toString(){
return verb + "(" + object + ", " + subject + ")";
}
}

哈希映射构造：

public class Ontology {
private List<Sentence> sentences = new ArrayList<>();
/*
 * The following maps store the relation of a string occurring
 * as a subject or object, respectively, to the list of Sentence
 * ordinals where they occur.
 */
private Map<String,List<Integer>> subject2index = new HashMap<>();
private Map<String,List<Integer>> object2index = new HashMap<>();
/*
 * This set contains strings that occur as both,
 * subject and object. This is useful for determining strings
 * acting as an in-between connecting two relations. 
 */
private Set<String> joints = new HashSet<>();
public void addSentence( Sentence s ){
// add Sentence to the list of all Sentences
sentences.add( s );
// add the Subject of the Sentence to the map mapping strings
// occurring as a subject to the ordinal of this Sentence
List<Integer> subind = subject2index.get( s.getSubject() );
if( subind == null ){
   subind = new ArrayList<>();
    subject2index.put( s.getSubject(), subind );
}
subind.add( sentences.size() - 1 );
// add the Object of the Sentence to the map mapping strings
// occurring as an object to the ordinal of this Sentence
List<Integer> objind = object2index.get( s.getObject() );
if( objind == null ){
    objind = new ArrayList<>();
    object2index.put( s.getObject(), objind );
}
objind.add( sentences.size() - 1 );
// determine whether we've found a "joining" string
if( subject2index.containsKey( s.getObject() ) ){
    joints.add( s.getObject() );
}
if( object2index.containsKey( s.getSubject() ) ){
    joints.add( s.getSubject() );
}
}
public Collection<String> getJoints(){
return joints;
}
public List<Integer> getSubjectIndices( String subject ){
return subject2index.get( subject );
}
public List<Integer> getObjectIndices( String object ){
return object2index.get( object );
}
public Sentence getSentence( int index ){
return sentences.get( index );
}
}

最后是确定规则的代码：

public static void main(String[] args) throws IOException {
Ontology ontology = new Ontology();
BufferedReader br = new BufferedReader(new FileReader("file.txt"));
Pattern p = Pattern.compile("'(.*?)'\('(.*?)','(.*?)'\)"); 
String line;
while ((line = br.readLine()) != null) {
    Matcher m = p.matcher(line);
    if( m.matches() ) {
        String verb    = m.group(1);
        String object  = m.group(2);
        String subject = m.group(3);
        ontology.addSentence( new Sentence( verb, object, subject ) );
    }
}

for( String joint: ontology.getJoints() ){
    for( Integer subind: ontology.getSubjectIndices( joint ) ){
        Sentence xaS = ontology.getSentence( subind );
        for( Integer obind: ontology.getObjectIndices( joint ) ){
            Sentence yOb = ontology.getSentence( obind );
            Sentence s = new Sentence( xaS.getVerb(),
                                       xaS.getObject(),
                                       yOb.getSubject() );
            System.out.println( s );
        }
    }
}
}

是否有某种快速有效的方法可以从该哈希映射中消除重复项，为每个唯一规则只保留一个实例，同时将新索引与我们观察到的该规则的相同实例数相关联原图？

我想在处理完句子后消除重复的'rules'。但只有在我有机会计算每条规则出现的频率并将其保存为与我最终保留的唯一规则相关联的值之后。

Answer 1

我建议对您的数据模型进行一些更改。您可以很容易地存储一个句子在 Map 中出现的次数，如下所示：

Map<Sentence, Integer> sentenceCount = new HashMap<>();

这依赖于为 Sentence 实现 equals 和 hashCode 方法。它通过使用 Sentence 作为键自动消除重复项。

您可以按如下方式添加新句子：

public addSentence(Sentence sentence) {
    if (!sentenceCount.containsKey(sentence))
        sentenceCount.put(sentence, 0);
    sentenceCount.put(sentence, sentenceCount.get(sentence) + 1);
}

现在您不再需要 sentences 列表，因为您可以使用 sentenceCount.keySet().

获取句子集

如果您需要从主语和宾语到句子的映射，那么我不建议您使用索引：这是一种容易出错的方法。相反，我建议您制作直接地图：

Map<String, Set<Sentence>> subjectMap;
Map<String, Set<Sentence>> objectMap;

您可以使用它来查找某个主题出现的次数：

subjectMap.get("subject").stream().mapToInt(sentenceCount::get).sum();

Answer 2

如果您能够使用 Guava，则可以使用 Multiset 的实现。用户指南中的示例听起来与您的要求相当相似。

哈希映射，删除重复项，但也存储原始事件的数量。 java

hash map, remove duplicates but also store # of original occurances. java

java

hashmap

duplicate-removal