如何限制 Sphinx 使用内存？

Question

我的系统是

% uname -or
FreeBSD 11.0-RELEASE-p2

Sphinx 版本为

% searchd --help
Sphinx 2.2.11-id64-release (95ae9a6)

狮身人面像配置：

index content_rt_template : common_template
{
    type              = rt
    rt_mem_limit      = 128M              # 128M only...

    rt_field          = text
    rt_attr_string    = text
    rt_field          = title
    rt_attr_string    = title
    rt_field          = url
    rt_attr_string    = url
    rt_attr_bigint    = item_id
    rt_attr_uint      = source_id
    rt_attr_timestamp = published_date
    rt_attr_timestamp = created_date
}

common {
    lemmatizer_base   = /path/to/sphinx/
}

indexer
{
    mem_limit         = 128M              # 128M only...
}


index content_rt_from_20170501_to_20170601 : content_rt_template
{
    path              = /path/to/sphinx/data/2017/content_rt_from_20170501_to_20170601
}

index content_rt_from_20170601_to_20170701 : content_rt_template
{
    path              = /path/to/sphinx/data/2017/content_rt_from_20170601_to_20170701
}

index content_rt_from_20171201_to_20180101 : content_rt_template
{
    path              = /path/to/sphinx/data/2017/content_rt_from_20171201_to_20180101
}

index content2017
{
    type              = distributed
    local             = content_rt_from_20170501_to_20170601
    local             = content_rt_from_20170601_to_20170701
    local             = content_rt_from_20171201_to_20180101
}

searchd
{
    listen            = 127.0.0.1:9417
    listen            = 9317:mysql41
    log               = /path/to/sphinx/log/searchd_2017.log
    query_log         = /path/to/sphinx/log/query_2017.log
    read_timeout      = 60
    max_children      = 30
    pid_file          = /path/to/sphinx/pid/searchd2017.pid
    seamless_rotate   = 0
    preopen_indexes   = 0
    unlink_old        = 1
    workers           = threads # for RT to work
    binlog_path       = /path/to/sphinx/data/2017/
}

启动 Sphinx 前使用的内存：

Mem[|||||||||                     5.33G/40.0G]
Swp[||||||||||||||||||||||||||||||3.35G/4.00G]

登录 Sphinx 开始：

% ./start.sh
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/path/to/sphinx/conf/content2017.conf'...
listening on 127.0.0.1:9417
listening on all interfaces, port=9317
WARNING: index 'common_template': key 'path' not found - NOT SERVING
WARNING: index 'content_rt_template': path must be specified - NOT SERVING
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20170501_to_20170601'
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20170601_to_20170701'
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_rt_from_20171201_to_20180101'
precached 3 indexes in 6.520 sec
Sphinx 2.2.11-id64-release (95ae9a6)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file '/path/to/sphinx/conf/content_dist.conf'...
listening on 127.0.0.1:9312
listening on all interfaces, port=9306
WARNING: index 'common_template': key 'path' not found - NOT SERVING
WARNING: index 'content_rt_template': path must be specified - NOT SERVING
WARNING: failed to init process shared rwlock: process shared rwlock is not supported by FreeBSD; ALTER disabled
precaching index 'content_snippet'
precached 1 indexes in 0.064 sec

Sphinx 启动后使用的内存：

Mem[|||||||||||||||               11.6G/40.0G]
Swp[||||||||||||||||||||||||||||||3.34G/4.00G]

Sphinx 使用大约6G。但是根据mem_limit和rt_mem_limit必须使用不超过128*3=384M。

使用大量内存的原因可能是什么？

可能是这个警告的原因？

WARNING: failed to init process shared rwlock: process shared rwlock

UPD

我已经在 Ubuntu 16.04 上试过了，情况是一样的。

Answer 1

有选项

ondisk_attrs = pool

RT 索引使用的 RAM 不多。

Answer 2

rt_mem_limit只限制了ram块本身的大小。任何磁盘块都将使用自己的内存。

通常情况下，构成最大部分的属性都保存在内存中。可以用 http://sphinxsearch.com/docs/current.html#conf-ondisk-attrs

减少内存

如何限制 Sphinx 使用内存？

How to limit using memory by Sphinx?

freebsd

sphinx