将日志从 fluentd 发送到 splunk

Question

我正在使用 log4j ，所以有不同格式的日志。我能够使用以下多行格式将大部分日志从 fluentd 发送到 splunk，但其中很少有不同的行为（日期格式不同的日志）。

<source>
  @type tail
  path /tmp/LOG_SPLUNK.*
  pos_file /tmp/my-splunk.pos
  path_key log_type
  read_from_head true
  tag "splunk.#log.mylogs"
  format multiline
  format_firstline /^\[/
  format1 /\[(?<timestamp>[^ ]* [^ ]*)\] (?<level>[^ ]*) (?<message>.*)/
  time_type string
  time_key timestamp
  time_format %Y-%m-%d %H:%M:%S,%N
  keep_time_key true
</source>

以下是日志格式：

[2022-04-13 06:27:08,340] INFO Loading plugin from: /my/path (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
java.lang.NullPointerException
    at java.util.Properties$LineReader.readLine(Properties.java:434)
    at java.util.Properties.load0(Properties.java:353)
    at java.util.Properties.load(Properties.java:341)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:380)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.versionFor(DelegatingClassLoader.java:385)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:355)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:328)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:261)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:253)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:222)
    at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:199)
    at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:60)
    at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:91)
    at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
[2022-04-13 06:27:09,520] INFO Registered loader: PluginClassLoader{pluginLocation=file:/my/path/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
Apr 13, 2022 6:27:17 AM org.glassfish.jersey.internal.inject.Providers checkProviderRuntime
WARNING: A provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource will be ignored.

我可以将上述所有格式发送到 splunk，但有些格式不同。有没有我可以处理所有格式的格式。如果我收到模式不匹配错误，我可以包含格式，但我没有

Answer 1

试试这个。

[(?[^]* [^])] (?[^]) (?[\W\w]+)

.* 在新行停止。 [\W\w]+ 将捕获消息字段中的整个堆栈跟踪。

将日志从 fluentd 发送到 splunk

Sending logs from fluentd to splunk

regex

multiline

splunk

fluentd

splunk-query