Elasticsearch 6.5 亿记录优化

Question

如果我的记录每年可以扩展到 6.58 亿条记录，我会尝试找出最适合 elasticsearch 的解决方案。实际上，现在我的所有记录都有一个索引，有 2 个分片和 0 个副本。此外，我注意到使用 356k 条记录和一个索引，排序和搜索比具有 1000 条记录的 365 个索引更快。问题是，如果我要对 6.58 亿条记录进行排序搜索并删除超过一年的记录或索引，最好和更快的存储数据的方法是什么？

Elasticsearch 6.2 版本，javascriptapi。

const defaultPageSize = 10
const indexTemplateSettings = {
    number_of_shards: 2,
    number_of_replicas : 0,
    max_result_window: 1000000000,
    'index.routing.allocation.enable': 'all',
}

const createClient = () =>
    new elasticsearch.Client({
        host: `${config.elastic.host}:${config.elastic.port}`,
        log: config.elastic.logLevel,
        httpAuth: `${config.elastic.userName}:${config.elastic.password}`,
    })

export const get = ({index, skip = 0, pageSize = defaultPageSize, search, sort = {by: 'timestamp', direction: 'desc'}}) => new Promise(async resolve => {
    try {
        logger.silly(`getting data from elastic: index: ${index}, skip: ${skip}, pageSize: ${pageSize}`)

        let client = createClient()

        const sortSettings = {
            order: `${sort.direction.toLowerCase()}`,
            missing: '_last',
            unmapped_type: 'long',
        }

        const params = {
            from: skip,
            size: pageSize || undefined,
            index: `${index.toLowerCase()}`,
            filter_path: 'hits.hits._source, hits.total',
            body: {
                query: {'match_all': {}},
                sort: {
                    [`${sort.by}.keyword`]: sortSettings,
                    [`${sort.by}.seconds`]: sortSettings,
                },
            },
        }

        if (search) {
            params.body.query = {
                query_string : {
                    query: `*${search}* OR *${search}`,
                    analyze_wildcard: true,
                },
            }
        }

        await client.search(params,
            (e, {hits: {hits:  data = [], total: totalCount} = {hits: [], total: 0}} = {}) => {
                logger.silly(`elastic searching completed. Result: contains ${totalCount} items`)

                resolve({items: data.map(t => t._source), totalCount})
            })
    } catch (e) {
        logger.error(e)
    }
})

export const push = (message, type) => new Promise(async resolve => {
    try {
        let client = createClient()
        let oneYearAgoTime = new Date(new Date().setFullYear(new Date().getFullYear() - 1)).toISOString().substring(0, 10)
        let indexCreationTime = new Date('2016-04-27').toISOString().substring(0, 10)

           await client.deleteByQuery({
            index: type.toLowerCase(),
            body: {
                query: {
                    range: {
                        '_timestampIndex' : {
                            lte: oneYearAgoTime,
                        },
                    },
                },
            },
        } , (error, response) => {
            logger.silly('Deleted of data completed', response)
        })

        await client.index({
            index: type.toLowerCase(),
            type,
            body: {
                ...message,
                _timestampIndex: indexCreationTime,
            },
        },
        (error, response) => {
            logger.silly('Pushing of data completed', response)

            resolve(response)
        })

    } catch (e) {
        logger.error(e)
    }
})

Answer 1

每个分片 1,000 个文档太少了。根据经验，分片应该在 GB 范围内；取决于用例，介于 10GB（搜索）到 50GB（日志）之间——假设您有一台大小合适的机器。如果我在您的评论中看到正确的话，您有 160 万个文档，需要 333MB 的存储空间 space。因此，您将拥有大约 400 倍的文档，也就是大约 133GB 的数据；也许10个碎片？如果您想对其进行适当的基准测试，请使用 1 个分片并查看它何时爆炸 — 这应该让您了解最大分片大小。
从索引中删除文档总是代价高昂。 Time-based 索引（如果您的分片足够大）或过滤器（甚至 filtered alias 适合的时间）可能会让您避免频繁删除大量文档。

Elasticsearch 6.5 亿记录优化

Elasaticsearch 650 million records optimization

javascript

node.js

elasticsearch

elasticsearch-5

elasticsearch-6