Virtuoso：当我得到 'Checkpoint removed ** MB of remapped pages, leaving * MB' 时会发生什么

Question

我正在将 freebase 文件加载到 virtuoso 中。我有 88 个文件夹。每个文件夹包含 4 个文件。我这样划分文件是因为我只有 8 GB RAM。

#!/bin/bash
for i in {1..88}
do
  isql 1111 dba dba exec="ld_dir('/data/data/${i}', '*.nt', 'http://freebase.com');"
  isql 1111 dba dba exec="rdf_loader_run();" 
  isql 1111 dba dba exec="checkpoint;"
  isql 1111 dba dba exec="commit WORK;"
  isql 1111 dba dba exec="checkpoint;"
  isql 1111 dba dba exec="delete from DB.DBA.load_list;"
done

Virtuoso.ini

# each buffer caches a 8K page of data and occupies approx. 8700 bytes of memory
# it's suggested to set this value to 65 % of ram for a db only server
# so if you have 32 GB of ram: 32*1000^3*0.65/8700 = 2390804
# default is 2000 which will use 16 MB ram
[Database]
MaxCheckpointRemap = 150000 (I have 8GB of RAM)
[TempDatabase]
MaxCheckpointRemap      = 2000
NumberOfBuffers          = 170000
MaxDirtyBuffers          = 130000

留言

Checkpoint 删除了 628 MB 的重新映射页面，留下 31 MB。持续时间 31.21 秒。为了节省这个时间，增加 MaxCheckpointRemap and/or 在 ini 文件中将 Unremap quota 设置为 0。

问题

为什么我会收到这条消息？是否影响数据库的加载过程和构建？

更新

    #!/bin/bash
   
    #clear list
    isql 1111 dba dba exec="delete from DB.DBA.load_list;"
    #load data
    isql 1111 dba dba exec="ld_dir('/data/data, '*.gz', 'http://freebase.com');"
    isql 1111 dba dba exec="set isolation='uncommitted';"
    isql 1111 dba dba exec="rdf_loader_run();" 
    #checkpoint
    isql 1111 dba dba exec="checkpoint;"
    isql 1111 dba dba exec="commit WORK;"
    isql 1111 dba dba exec="checkpoint;"

virtuoso.ini

MaxCheckpointRemap       = 150000 (8GB RAM)
# Uncomment next two lines if there is 4 GB system memory free
NumberOfBuffers          = 340000
MaxDirtyBuffers          = 250000
# I have 6 cores
ThreadsPerQuery         = 4
AsyncQueueMaxThreads        = 10
#NO THREADS LEFT IN THE QUEUE
ThreadCleanupInterval       = 0
ThreadThreshold         = 0
#check point every 30 minutes.
CheckpointInterval      = 30

Answer 1

最好在 virtuoso docs 中解释，但简而言之：

The MaxCheckpointRemap parameter in the virtuoso.ini file controls how many pages may be stored on a page other than their logical page.

这意味着增加参数将节省您在检查点上的工作（和时间），但会降低数据局部性特征（一起插入的数据会保存在一起）。

Virtuoso：当我得到 'Checkpoint removed ** MB of remapped pages, leaving * MB' 时会发生什么

Virtuoso: What happens when I get 'Checkpoint removed ** MB of remapped pages, leaving * MB'

sparql

freebase

virtuoso

Virtuoso.ini

留言

问题

更新

virtuoso.ini