为什么我使用 "arq" 查询本地 rdf 文件时得到 "GC overhead limit exceeded"

Question

我正在使用 ARQ 来查询本地 RDF 文件。我使用的命令如下：

./arq --data /home/datasets/a-m-00027.nt --results CSV --query myQuery.sparql

myQuery.sparql 包含查询：

PREFIX basekb:<http://rdf.basekb.com/ns/>
PREFIX rdf:<http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT ?x
FROM  </home/data/a-m-00027.nt>
WHERE {?x rdf:type basekb:music.release} 
LIMIT 10

异常

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.concurrent.CopyOnWriteArrayList.iterator(CopyOnWriteArrayList.java:959)
        at com.hp.hpl.jena.graph.impl.SimpleEventManager.notifyAddTriple(SimpleEventManager.java:97)
        at com.hp.hpl.jena.graph.impl.GraphBase.notifyAdd(GraphBase.java:124)
        at com.hp.hpl.jena.graph.impl.GraphBase.add(GraphBase.java:203)
        at com.hp.hpl.jena.sparql.core.DatasetGraphCollection.add(DatasetGraphCollection.java:43)
        at com.hp.hpl.jena.sparql.core.DatasetGraphBase.add(DatasetGraphBase.java:82)
        at org.apache.jena.riot.system.StreamRDFLib$ParserOutputDataset.triple(StreamRDFLib.java:206)
        at org.apache.jena.riot.lang.LangNTriples.runParser(LangNTriples.java:61)
        at org.apache.jena.riot.lang.LangBase.parse(LangBase.java:42)
        at org.apache.jena.riot.RDFParserRegistry$ReaderRIOTLang.read(RDFParserRegistry.java:185)
        at org.apache.jena.riot.RDFDataMgr.process(RDFDataMgr.java:906)
        at org.apache.jena.riot.RDFDataMgr.parse(RDFDataMgr.java:687)
        at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:534)
        at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:501)
        at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:454)
        at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:432)
        at org.apache.jena.riot.RDFDataMgr.read(RDFDataMgr.java:422)
        at arq.cmdline.ModDatasetGeneral.addGraphs(ModDatasetGeneral.java:98)
        at arq.cmdline.ModDatasetGeneral.createDataset(ModDatasetGeneral.java:87)
        at arq.cmdline.ModDatasetGeneralAssembler.createDataset(ModDatasetGeneralAssembler.java:35)
        at arq.cmdline.ModDataset.getDataset(ModDataset.java:34)
        at arq.query.getDataset(query.java:176)
        at arq.query.queryExec(query.java:198)
        at arq.query.exec(query.java:159)
        at arq.cmdline.CmdMain.mainMethod(CmdMain.java:102)
        at arq.cmdline.CmdMain.mainRun(CmdMain.java:63)
        at arq.cmdline.CmdMain.mainRun(CmdMain.java:50)
        at arq.arq.main(arq.java:28)

实例

<http://rdf.basekb.com/ns/architecture.building_complex>        <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>       <http://rdf.basekb.com/ns/type.type>

是否正在将整个文件加载到内存中？

Answer 1

因为异常告诉您内存不足：

java.lang.OutOfMemoryError: GC overhead limit exceeded

很可能您实际上并没有内存不足，而只是您的 JVM 设置默认情况下不会超过一定的内存量。如中所述尝试运行

JVM_ARGS="-Xmx4096M" ./arq --data /home/datasets/a-m-00027.nt --results CSV --query myQuery.sparql

Answer 2

Is the whole file being loaded into memory?

没错，这是您的问题。如前所述，您可以增加 java 堆并使其适合。

但作为替代方案，或者在您根本没有足够内存的情况下，请尝试使用 TDB 来存储和索引文件然后查询它：

$ tdbloader --loc my_tdb_store /home/datasets/a-m-00027.nt
$ tdbquery --loc my_tdb_store --results CSV --query myQuery.sparql

（完成后可以删除存储，它只是一个名为my_tdb_store的目录）

作为第三种选择，您可以完全跳过 sparql。你查询只找到类型为basekb:music.release的前十个东西，你可以这样找到：

$ riot /home/datasets/a-m-00027.nt | \
  grep '<http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://rdf.basekb.com/ns/music.release> .' | \
  cut -d ' ' -f 1 | \
  head -10

使用最少的内存。

为什么我使用 "arq" 查询本地 rdf 文件时得到 "GC overhead limit exceeded"

Why am I getting "GC overhead limit exceeded" when I use "arq" to query local rdf files

rdf

sparql

freebase

异常

实例