当 "time-unit" = "days" 时，ELK Curator 如何管理 monthly/weekly 索引

Question

我们在公司的标准 elasticsearch 和 curator 实施之上有一个自定义包装器。我想知道当默认 "time-unit" 设置为 "days".

时，策展人处理 "Monthly/Weekly" 索引的行为是什么

**我无法覆盖默认值 "time-unit"

这是我们的 monthly/weekly 索引命名方式的示例格式

月度指数格式

logstash-test-monthly-2018.01
logstash-test-monthly-2018.02
logstash-test-monthly-2018.03
logstash-test-monthly-2018.04
...
...
logstash-test-monthly-2018.12

每周索引的格式

logstash-test-weekly-2018.01
logstash-test-weekly-2018.02
...
...
...
logstash-test-weekly-2018.51
logstash-test-weekly-2018.52

Delete_Index.yml - 馆长删除说明

actions:
  1:
    action: delete_indices
    options:
      ignore_empty_list: true
    filters:
      - exclude: true
        filtertype: kibana
      - exclude: false
        kind: regex
        filtertype: pattern
        value: .*-monthly-.*
      - range_to: 0
        filtertype: period
        source: name
        range_from: -60
        period_type: relative
        timestring: '%Y.%m.%d'
        exclude: true
        unit: days
    description: Delete indices more than X days old
  2:
    action: delete_indices
    options:
      ignore_empty_list: true
    filters:
      - exclude: true
        filtertype: kibana
      - exclude: false
        kind: regex
        filtertype: pattern
        value: .*-weekly-.*
      - range_to: 0
        filtertype: period
        source: name
        range_from: -30
        period_type: relative
        timestring: '%Y.%m.%d'
        exclude: true
        unit: days

实施上述配置，每月索引保留 60 天，每周索引保留 30 天。

配置执行于**2018 年 4 月 4 日，结果为**

执行后保留的每月索引

logstash-test-monthly-2018.03
logstash-test-monthly-2018.04

由于上述索引^^仅包含 31+4=35 天的索引数据，而不是预期的 60 天。

我期待馆长会保留以下索引

logstash-test-monthly-2018.02
logstash-test-monthly-2018.03
logstash-test-monthly-2018.04

谁能解释为什么 curator 无法保留 60 天的索引？

Answer 1

TL;DR:二月份天数比较短，age计算是秒的倍数*合适的units.

所有这些都在 Elastic 网站的 age filter documentation 中进行了解释。

age filter vs. period filter

The time differential means of calculation can lead to frustration.

Setting unit to months, and unit_count to 3 will actually calculate the age as 3*30*24*60*60, which is 7776000 seconds. This may be a big deal. If the date is 2017-01-01T02:30:00Z, or 1483237800 in epoch time, subtracting 7776000 seconds makes 1475461800, which is 2016-10-03T02:30:00Z. If you were to try to match monthly indices, index-2016.12, index-2016.11, 2016.10, 2016.09, etc., then both index-2016.09 and index-2016.10 will be older than the cutoff date. This may result in unintended behavior.

Another way this can cause issues is with weeks. Weekly indices may start on Sunday or Monday. The age filter’s calculation doesn’t take this into consideration, and merely tests the difference between execution time and the timestamp on the index (from any source).

Another means of selecting indices and snapshots is the period filter, which is perhaps a better choice for selecting weeks and months as it compensates for these differences.

一旦您了解 age 的计算只不过是乘以 unit_count * unit 的适当秒数，那么留存率会以这种方式发生是有道理的这是。如前所述，使用 period 过滤器可能会做得更好，因为它适用于完整的日、周、月和年。

Answer 2

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 2 months (based on index name), for custom-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      continue_if_exception: False
      disable_action: False
    filters:
      - filtertype: pattern
        kind: regex
        value: ^(index-pattern).*$
      - filtertype: age
        source: name
        direction: older
        timestring: "%Y.%m"
        unit: months
        unit_count: 2

这是仅适用于删除每月索引的 yml 配置。

当 "time-unit" = "days" 时，ELK Curator 如何管理 monthly/weekly 索引

How does ELK Curator manage monthly/weekly indexes when the "time-unit" = "days"

elasticsearch-curator

elasticsearch-5