Duke 重复数据删除引擎:完全相同的记录不匹配
Duke deduplication engine : exact same record not matched
我正在尝试使用 Duke 将记录从一个 csv 匹配到 another.First csv,第二个都有 ID,Model,Price,CompanyName,Review,Url
列。我正在尝试与另一个 csv 匹配以查找重复记录。
package no.priv.garshol.duke;
import no.priv.garshol.duke.matchers.PrintMatchListener;
public class RunDuke {
public static void main(String[] argv) throws Exception {
Configuration config =
ConfigLoader
.load("/home/kishore/Duke-master/doc/example-data/presonalCare.xml");
Processor proc = new Processor(config);
proc.addMatchListener(new PrintMatchListener(true, true, true, false, config.getProperties(),
true));
proc.link();
proc.close();
}
}
这里是 personalCare.xml
的例子:
<!-- language: xml -->
<!-- For more information, see https://github.com/larsga/Duke/wiki/ Improvements
needed: - some area numbers have spaces in them - not stripping accents from
names -->
<duke>
<schema>
<threshold>0.7</threshold>
<property type="id">
<name>ID</name>
</property>
<property>
<name>Model</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.4</low>
<high>0.8</high>
</property>
<property>
<name>Price</name>
<comparator>no.priv.garshol.duke.comparators.ExactComparator</comparator>
<low>0.04</low>
<high>0.73</high>
</property>
<property>
<name>CompanyName</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.4</low>
<high>0.8</high>
</property>
<property>
<name>Review</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.12</low>
<high>0.93</high>
</property>
<property>
<name>Url</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.12</low>
<high>0.93</high>
</property>
</schema>
<database class="no.priv.garshol.duke.databases.InMemoryDatabase">
</database>
<group>
<csv>
<param name="input-file" value="personal_care_11.csv" />
<param name="header-line" value="false" />
<column name="1" property="ID" />
<column name="2" property="Model" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="3" property="Price" />
<column name="4" property="CompanyName" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="5" property="Review" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="6" property="Url" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
</csv>
</group>
<group>
<csv>
<param name="input-file" value="personal_care_11.csv" />
<param name="header-line" value="false" />
<column name="1" property="ID" />
<column name="2" property="Model" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="3" property="Price" />
<column name="4" property="CompanyName" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="5" property="Review" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="6" property="Url" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
</csv>
</group>
</duke>
上面的代码工作正常,但它与确切的记录示例不匹配
STHDRNFKAQ4AFYE8,Littmann 3M Classic II S.E Acoustic
Stethoscope,6297,Littmann,,http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=STHDRNFKAQFAFYE8&affid=3ba0de4902524e2b90e43b84b89ea0ef
这两个 csv 文件中都有。我还想知道 .xml
文件中给出的低值和高值 属性 的工作,如何确定列值的低值和高值。
您正在进行记录链接(两个数据集)而不是重复数据删除(单个数据集),所以去掉 .deduplicate() 调用。
此外,请不要使用 'no.priv.garshol.duke' 包名。您永远不应该使用不属于您自己的域名。
无论如何,您找不到任何匹配项的原因是这两条记录具有相同的 ID。 Duke 验证它没有将记录报告为匹配自身,因此匹配被过滤掉。如果您复制 csv 文件并将其用于第 2 组,然后更改 ID,Duke 会找到重复项。
以下是我尝试这样做时发生的情况:
[lars.garshol@laptop tmp]$ java -cp ~/cvs-co/duke/duke-core/target/duke-core-1.3-SNAPSHOT.jar:. no.priv.garshol.duke.Duke --showmatches presonalCare.xml
MATCH 0.9982630751840313
ID: 'SHDRNFKAQ4AFYE8', Model: 'littmann 3m classic ii s.e acoustic stethoscope', Price: '6297', CompanyName: 'littmann', Url: 'http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=sthdrnfkaqfafye8&affid=3ba0de4902524e2b90e43b84b89ea0ef',
ID: 'STHDRNFKAQ4AFYE8', Model: 'littmann 3m classic ii s.e acoustic stethoscope', Price: '6297', CompanyName: 'littmann', Url: 'http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=sthdrnfkaqfafye8&affid=3ba0de4902524e2b90e43b84b89ea0ef',
我正在尝试使用 Duke 将记录从一个 csv 匹配到 another.First csv,第二个都有 ID,Model,Price,CompanyName,Review,Url
列。我正在尝试与另一个 csv 匹配以查找重复记录。
package no.priv.garshol.duke;
import no.priv.garshol.duke.matchers.PrintMatchListener;
public class RunDuke {
public static void main(String[] argv) throws Exception {
Configuration config =
ConfigLoader
.load("/home/kishore/Duke-master/doc/example-data/presonalCare.xml");
Processor proc = new Processor(config);
proc.addMatchListener(new PrintMatchListener(true, true, true, false, config.getProperties(),
true));
proc.link();
proc.close();
}
}
这里是 personalCare.xml
的例子:
<!-- language: xml -->
<!-- For more information, see https://github.com/larsga/Duke/wiki/ Improvements
needed: - some area numbers have spaces in them - not stripping accents from
names -->
<duke>
<schema>
<threshold>0.7</threshold>
<property type="id">
<name>ID</name>
</property>
<property>
<name>Model</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.4</low>
<high>0.8</high>
</property>
<property>
<name>Price</name>
<comparator>no.priv.garshol.duke.comparators.ExactComparator</comparator>
<low>0.04</low>
<high>0.73</high>
</property>
<property>
<name>CompanyName</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.4</low>
<high>0.8</high>
</property>
<property>
<name>Review</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.12</low>
<high>0.93</high>
</property>
<property>
<name>Url</name>
<comparator>no.priv.garshol.duke.comparators.Levenshtein</comparator>
<low>0.12</low>
<high>0.93</high>
</property>
</schema>
<database class="no.priv.garshol.duke.databases.InMemoryDatabase">
</database>
<group>
<csv>
<param name="input-file" value="personal_care_11.csv" />
<param name="header-line" value="false" />
<column name="1" property="ID" />
<column name="2" property="Model" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="3" property="Price" />
<column name="4" property="CompanyName" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="5" property="Review" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="6" property="Url" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
</csv>
</group>
<group>
<csv>
<param name="input-file" value="personal_care_11.csv" />
<param name="header-line" value="false" />
<column name="1" property="ID" />
<column name="2" property="Model" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="3" property="Price" />
<column name="4" property="CompanyName" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="5" property="Review" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
<column name="6" property="Url" cleaner="no.priv.garshol.duke.cleaners.LowerCaseNormalizeCleaner"/>
</csv>
</group>
</duke>
上面的代码工作正常,但它与确切的记录示例不匹配
STHDRNFKAQ4AFYE8,Littmann 3M Classic II S.E Acoustic
Stethoscope,6297,Littmann,,http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=STHDRNFKAQFAFYE8&affid=3ba0de4902524e2b90e43b84b89ea0ef
这两个 csv 文件中都有。我还想知道 .xml
文件中给出的低值和高值 属性 的工作,如何确定列值的低值和高值。
您正在进行记录链接(两个数据集)而不是重复数据删除(单个数据集),所以去掉 .deduplicate() 调用。
此外,请不要使用 'no.priv.garshol.duke' 包名。您永远不应该使用不属于您自己的域名。
无论如何,您找不到任何匹配项的原因是这两条记录具有相同的 ID。 Duke 验证它没有将记录报告为匹配自身,因此匹配被过滤掉。如果您复制 csv 文件并将其用于第 2 组,然后更改 ID,Duke 会找到重复项。
以下是我尝试这样做时发生的情况:
[lars.garshol@laptop tmp]$ java -cp ~/cvs-co/duke/duke-core/target/duke-core-1.3-SNAPSHOT.jar:. no.priv.garshol.duke.Duke --showmatches presonalCare.xml
MATCH 0.9982630751840313
ID: 'SHDRNFKAQ4AFYE8', Model: 'littmann 3m classic ii s.e acoustic stethoscope', Price: '6297', CompanyName: 'littmann', Url: 'http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=sthdrnfkaqfafye8&affid=3ba0de4902524e2b90e43b84b89ea0ef',
ID: 'STHDRNFKAQ4AFYE8', Model: 'littmann 3m classic ii s.e acoustic stethoscope', Price: '6297', CompanyName: 'littmann', Url: 'http://dl.flipkart.com/dl/littmann-3m-classic-ii-s-e-acoustic-stethoscope/p/itme3uhzbqxhzfda?pid=sthdrnfkaqfafye8&affid=3ba0de4902524e2b90e43b84b89ea0ef',