PHP 堆栈跟踪的 Filebeat 多行模式

Question

我正在尝试将 PHP FPM 日志导入 ELK 堆栈。为此，我使用 filebeat 来读取文件。在将此数据发送到 logstash 之前，应合并多行日志条目。

为此，我构建了这个 filebeat 配置：

filebeat.inputs:

  # Each - is an input. Most options can be set at the input level, so
  # you can use different inputs for various configurations.
  # Below are the input specific configurations.

  - type: filestream

    # Change to true to enable this input configuration.
    enabled: true

    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      - '/var/log/app/fpm/*.log'

    multiline.type: pattern
    multiline.pattern: '^\[\d{2}-\w{3}-\d{4} \d{2}:\d{2}:\d{2} [\w/]*\] PHP\s*at.*'
    multiline.negate: false
    multiline.match: after

    processors:
      - add_fields:
          fields.docker.service: "fpm"

但是正如您在 logstash 的 ruby 调试输出中看到的那样，消息没有合并：

{
        "@timestamp" => 2021-08-10T13:54:10.149Z,
        "agent" => {
                 "version" => "7.13.4",
                "hostname" => "3cb76d7d4c7d",
                      "id" => "61dec25e-12ec-4a65-9f1f-ec72a5aa83ee",
            "ephemeral_id" => "631db0d8-60ad-4625-891c-3da09cb0a442",
                    "type" => "filebeat"
        },
             "input" => {
            "type" => "filestream"
        },
               "log" => {
            "offset" => 344,
              "file" => {
                "path" => "/var/log/app/fpm/error.log"
            }
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_grokparsefailure"
        ],
            "fields" => {
            "docker" => {
                "service" => "fpm"
            }
        },
          "@version" => "1",
           "message" => "[17-Jun-2021 13:07:56 Europe/Berlin] PHP [WARN] (/var/www/html/Renderer/RendererTranslator.php:92) - unable to translate type integer. It is not a string (/url.php)",
               "ecs" => {
            "version" => "1.8.0"
        }
    }
    {
      "input" => {
            "type" => "filestream"
        },
            "module" => "PHP IES\ServerException",
               "ecs" => {
            "version" => "1.8.0"
        },
          "@version" => "1",
               "log" => {
            "offset" => 73,
              "file" => {
                "path" => "/var/log/ies/fpm/error.log"
            }
        },
        "@timestamp" => 2021-06-17T11:10:41.000Z,
             "agent" => {
                 "version" => "7.13.4",
                "hostname" => "3cb76d7d4c7d",
                      "id" => "61dec25e-12ec-4a65-9f1f-ec72a5aa83ee",
            "ephemeral_id" => "631db0d8-60ad-4625-891c-3da09cb0a442",
                    "type" => "filebeat"
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied"
        ],
            "fields" => {
            "docker" => {
                "service" => "fpm"
            }
        },
           "message" => "core.login"
    }
    {
        "@timestamp" => 2021-08-10T13:54:10.149Z,
             "agent" => {
                 "version" => "7.13.4",
                "hostname" => "3cb76d7d4c7d",
                      "id" => "61dec25e-12ec-4a65-9f1f-ec72a5aa83ee",
            "ephemeral_id" => "631db0d8-60ad-4625-891c-3da09cb0a442",
                    "type" => "filebeat"
        },
               "ecs" => {
            "version" => "1.8.0"
        },
             "input" => {
            "type" => "filestream"
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_grokparsefailure"
        ],
            "fields" => {
            "docker" => {
                "service" => "fpm"
            }
        },
          "@version" => "1",
           "message" => "[17-Jun-2021 13:10:41 Europe/Berlin] PHP     at App\Module\ComponentModel\ComponentModel->doPhase(/var/www/html/Component/Container.php:348)",
               "log" => {
            "offset" => 204,
              "file" => {
                "path" => "/var/log/app/fpm/error.log"
            }
        }
    }

我用 Rubular 测试了正则表达式，它与堆栈跟踪消息匹配。

我做错了什么？

Answer 1

您需要将 multiline.negate 设置为真。

Answer 2

我没有调整filebeat配置，而是调整了应用的日志配置。现在写了JSON个文件，可以用filebeat轻松读取。这样就不再需要考虑换行了。

PHP 堆栈跟踪的 Filebeat 多行模式

Filebeat multiline pattern for PHP stack trace

logstash

filebeat