从不同的事件和分隔符中提取 Splunk 字段
Splunk field extractions from different events & delimiters
我的关键事件时间戳的 Splunk 日志格式如下：
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]
我想将所有关键事件时间戳提取到如下所示的 table 字段中，这样我就可以在它们之间进行区分等：
Hour Id StartTime EndTime beginThread endThread zipBefore zipAfter
18-nov-2020 11:00:00 126566 18-nov-2020 12:17:10.603227 18-nov-2020 12:36:53.094513 18-nov-2020 12:17:10.905782 18-nov-2020 12:24:22.628907 18-nov-2020 12:27:08.776174 18-nov-2020 12:36:52.718122
此外，我在日志中的最后一个事件有不同的情况、线程和时间戳，我需要根据分隔符分别提取，如下所示：
Case Thread StartTime Count EndTime
CASE1 6 18-nov-2020 12:17:11.377070 0 18-nov-2020 12:17:12.608526
CASE1 0 18-nov-2020 12:17:11.365409 0 18-nov-2020 12:17:12.654285
CASE2 5 18-nov-2020 12:17:13.295225 700000 18-nov-2020 12:23:29.370142
CASE2 2 18-nov-2020 12:17:12.815714 700000 18-nov-2020 12:23:31.400500
CASE3 0 18-nov-2020 12:17:12.644921 11 18-nov-2020 12:17:13.636655
下面是完成第一个任务的示例查询。然而，一个问题是它只适用于一组事件。由于8个事件之间没有明显的联系，所以没有将一笔交易的8与另一笔交易的8分开。
| makeresults | eval data="[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]" | eval data=split(data,"!") | mvexpand data | eval _raw=data
```The above just sets up test data```
| rex "\[Message=(?<Message>[^\]]+)"
| rex field=Message "(?<field>\w+)\s+=\s+(?<value>.*)"
| eval {field}=value
| table Hour Id StartTime EndTime beginThread endThread zipBefore zipAfter
| filldown | tail 1
第二部分与第一部分类似，但解析不同
| makeresults | eval data="[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]" | eval data=split(data,"!") | mvexpand data | eval _raw=data
```The above just sets up test data```
| rex "\[Message=(?<Message>[^\]]+)"
| rex field=Message "(?<field>\w+)\s+=\s+(?<value>.*)"
| eval {field}=value
```We only care about LogTime messages```
| search LogTime=*
```Divide the message on commas and make separate events```
| eval LogTime=split(LogTime, ",") | mvexpand LogTime
```Parse the events```
| rex field=LogTime "(?<Case>[^~]+)~(?<Thread>[^~]+)~(?<StartTime>[^~]+)~(?<Count>[^~]+)~(?<EndTime>[^~]+)(?:,|$)"
| table Case Thread StartTime Count EndTime
我的关键事件时间戳的 Splunk 日志格式如下：
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]
我想将所有关键事件时间戳提取到如下所示的 table 字段中，这样我就可以在它们之间进行区分等：
Hour Id StartTime EndTime beginThread endThread zipBefore zipAfter
18-nov-2020 11:00:00 126566 18-nov-2020 12:17:10.603227 18-nov-2020 12:36:53.094513 18-nov-2020 12:17:10.905782 18-nov-2020 12:24:22.628907 18-nov-2020 12:27:08.776174 18-nov-2020 12:36:52.718122
此外，我在日志中的最后一个事件有不同的情况、线程和时间戳，我需要根据分隔符分别提取，如下所示：
Case Thread StartTime Count EndTime
CASE1 6 18-nov-2020 12:17:11.377070 0 18-nov-2020 12:17:12.608526
CASE1 0 18-nov-2020 12:17:11.365409 0 18-nov-2020 12:17:12.654285
CASE2 5 18-nov-2020 12:17:13.295225 700000 18-nov-2020 12:23:29.370142
CASE2 2 18-nov-2020 12:17:12.815714 700000 18-nov-2020 12:23:31.400500
CASE3 0 18-nov-2020 12:17:12.644921 11 18-nov-2020 12:17:13.636655
下面是完成第一个任务的示例查询。然而，一个问题是它只适用于一组事件。由于8个事件之间没有明显的联系，所以没有将一笔交易的8与另一笔交易的8分开。
| makeresults | eval data="[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]" | eval data=split(data,"!") | mvexpand data | eval _raw=data
```The above just sets up test data```
| rex "\[Message=(?<Message>[^\]]+)"
| rex field=Message "(?<field>\w+)\s+=\s+(?<value>.*)"
| eval {field}=value
| table Hour Id StartTime EndTime beginThread endThread zipBefore zipAfter
| filldown | tail 1
第二部分与第一部分类似，但解析不同
| makeresults | eval data="[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Hour = 18-nov-2020 11:00:00]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=Id = 126566]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipBefore = 18-nov-2020 12:27:08.776174]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=zipAfter = 18-nov-2020 12:36:52.718122]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=StartTime = 18-nov-2020 12:17:10.603227]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=EndTime = 18-nov-2020 12:36:53.094513]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=beginThread = 18-nov-2020 12:17:10.905782]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=endThread = 18-nov-2020 12:24:22.628907]!
[Date=2020-11-18] [Time=12:36:53] [Mode=DEBUG] [Class=PrintUtil] [Line=557] [Message=LogTime = CASE1~6~18-nov-2020 12:17:11.377070~0~18-nov-2020 12:17:12.608526,CASE1~0~18-nov-2020 12:17:11.365409~0~18-nov-2020 12:17:12.654285,CASE3~0~18-nov-2020 12:17:12.644921~11~18-nov-2020 12:17:13.636655,CASE2~5~18-nov-2020 12:17:13.295225~700000~18-nov-2020 12:23:29.370142,CASE2~2~18-nov-2020 12:17:12.815714~700000~18-nov-2020 12:23:31.400500]" | eval data=split(data,"!") | mvexpand data | eval _raw=data
```The above just sets up test data```
| rex "\[Message=(?<Message>[^\]]+)"
| rex field=Message "(?<field>\w+)\s+=\s+(?<value>.*)"
| eval {field}=value
```We only care about LogTime messages```
| search LogTime=*
```Divide the message on commas and make separate events```
| eval LogTime=split(LogTime, ",") | mvexpand LogTime
```Parse the events```
| rex field=LogTime "(?<Case>[^~]+)~(?<Thread>[^~]+)~(?<StartTime>[^~]+)~(?<Count>[^~]+)~(?<EndTime>[^~]+)(?:,|$)"
| table Case Thread StartTime Count EndTime