使用多行和选项卡解析存储在 syslog-ng 系统中的 Windows 事件日志
Parsing Windows Event Logs stored in syslog-ng system with multi-lines and tabs
我正在尝试解析尚未使用 SNARE、NXlog 或 Adiscon 格式化的数百个 Windows 事件日志（我不确定他们如何将它们发送到系统日志服务器）。
我 运行 遇到的问题是确定 best/most 解析多行、多选项卡式 Windows 事件文件的有效方法。我没有相应的 EVTX 文件（它只是一个日志文件）。
我的目标是将每个事件都放在一行上，没有制表符，这样我就可以使用 grep 和 awk 更轻松地解析它。
tr -d "\n\r" < windows.log 将所有内容放在一行中（删除换行符），现在我需要去除制表符（制表符不如换行符重要）并想出一种每次都添加新行的方法在看到 "Jan 14" 之前。
使用 Python、Perl 或 Powershell 可能有更好的方法，但我在这方面的经验有限。
示例日志文件：
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Subject:
Security ID: S-1-5-21-3015042641-2194367929-112691256-2051
Account Name: SVCACCT
Account Domain: MYDOMAIN
Logon ID: 0xD7FC64F5
Logon Type: 3
This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159649 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Subject:
Security ID: S-1-5-21-3015042641-2194367929-112691256-12106
Account Name: SVCACCT2
Account Domain: MYDOMAIN
Logon ID: 0xD7FC600A
Logon Type: 3
This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
压缩日志文件示例：
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off. Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-2051 Account Name: SVCACCT Account Domain: MYDOMAIN Logon ID: 0xD7FC64F5 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off. Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-2051 Account Name: SVCACCT2 Account Domain: MYDOMAIN Logon ID: 0xD7FC64F5 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
首先，我们删除所有控制字符。然后我们搜索 "Jan 14" 并在它之前添加一个换行符。最后，我们用 -s 标志调用 tr，用单个字符替换重复字符的实例。我不太确定这有多有效，但它可能会让你开始。
tr -d "[:cntrl:]" < windows.log | sed 's/Jan 14/\'$'\n&/g' | tr -s " "
结果
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-2051 Account Name: SVCACCT Account Domain: MYDOMAIN Logon ID: 0xD7FC64F5Logon Type: 3This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159649 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-12106 Account Name: SVCACCT2 Account Domain: MYDOMAIN Logon ID: 0xD7FC600ALogon Type: 3This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
与其尝试将每条记录压缩到 1 行然后尝试对其进行分析，不如将每个 12 行块作为一条记录处理。例如：
$ cat tst.awk
{
gsub(/\r/,"")
gsub(/^[[:space:]]+|[[:space:]]+$/,"")
lineNr = (NR - 1) % 12 + 1
}
lineNr == 1 {
f["hd"] = [=10=]
}
lineNr ~ /[45679]/ {
tag = val = [=10=]
sub(/:.*/,"",tag)
sub(/[^:]+:[[:space:]]*/,"",val)
f[tag] = val
}
lineNr == 11 {
f["tl"] = [=10=]
for (tag in f) {
print tag, "=", f[tag]
}
print "-------"
}
.
$ awk -f tst.awk file
tl = This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Logon ID = 0xD7FC64F5
Logon Type = 3
Account Name = SVCACCT
Security ID = S-1-5-21-3015042641-2194367929-112691256-2051
hd = Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Account Domain = MYDOMAIN
-------
tl = This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Logon ID = 0xD7FC600A
Logon Type = 3
Account Name = SVCACCT2
Security ID = S-1-5-21-3015042641-2194367929-112691256-12106
hd = Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159649 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Account Domain = MYDOMAIN
-------
使用这种方法，您可以简单地通过名称引用每个字段以进行打印或分析。您可以扩展上面的内容以将第一行中的所有单独字段映射到单独的 tags/values 例如
lineNr==1 {
f["timestamp"] = " " " "
...
}
或使用正则表达式匹配或任何有意义的东西。一旦你[完成了上述操作，在脚本的其余部分分析或打印任何你喜欢的东西就变得非常简单了。
我正在尝试解析尚未使用 SNARE、NXlog 或 Adiscon 格式化的数百个 Windows 事件日志（我不确定他们如何将它们发送到系统日志服务器）。
我 运行 遇到的问题是确定 best/most 解析多行、多选项卡式 Windows 事件文件的有效方法。我没有相应的 EVTX 文件（它只是一个日志文件）。
我的目标是将每个事件都放在一行上，没有制表符，这样我就可以使用 grep 和 awk 更轻松地解析它。
tr -d "\n\r" < windows.log 将所有内容放在一行中（删除换行符），现在我需要去除制表符（制表符不如换行符重要）并想出一种每次都添加新行的方法在看到 "Jan 14" 之前。
使用 Python、Perl 或 Powershell 可能有更好的方法，但我在这方面的经验有限。
示例日志文件：
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Subject:
Security ID: S-1-5-21-3015042641-2194367929-112691256-2051
Account Name: SVCACCT
Account Domain: MYDOMAIN
Logon ID: 0xD7FC64F5
Logon Type: 3
This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159649 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Subject:
Security ID: S-1-5-21-3015042641-2194367929-112691256-12106
Account Name: SVCACCT2
Account Domain: MYDOMAIN
Logon ID: 0xD7FC600A
Logon Type: 3
This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
压缩日志文件示例：
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off. Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-2051 Account Name: SVCACCT Account Domain: MYDOMAIN Logon ID: 0xD7FC64F5 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off. Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-2051 Account Name: SVCACCT2 Account Domain: MYDOMAIN Logon ID: 0xD7FC64F5 Logon Type: 3 This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
首先，我们删除所有控制字符。然后我们搜索 "Jan 14" 并在它之前添加一个换行符。最后，我们用 -s 标志调用 tr，用单个字符替换重复字符的实例。我不太确定这有多有效，但它可能会让你开始。
tr -d "[:cntrl:]" < windows.log | sed 's/Jan 14/\'$'\n&/g' | tr -s " "
结果
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-2051 Account Name: SVCACCT Account Domain: MYDOMAIN Logon ID: 0xD7FC64F5Logon Type: 3This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159649 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.Subject: Security ID: S-1-5-21-3015042641-2194367929-112691256-12106 Account Name: SVCACCT2 Account Domain: MYDOMAIN Logon ID: 0xD7FC600ALogon Type: 3This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
与其尝试将每条记录压缩到 1 行然后尝试对其进行分析，不如将每个 12 行块作为一条记录处理。例如：
$ cat tst.awk
{
gsub(/\r/,"")
gsub(/^[[:space:]]+|[[:space:]]+$/,"")
lineNr = (NR - 1) % 12 + 1
}
lineNr == 1 {
f["hd"] = [=10=]
}
lineNr ~ /[45679]/ {
tag = val = [=10=]
sub(/:.*/,"",tag)
sub(/[^:]+:[[:space:]]*/,"",val)
f[tag] = val
}
lineNr == 11 {
f["tl"] = [=10=]
for (tag in f) {
print tag, "=", f[tag]
}
print "-------"
}
.
$ awk -f tst.awk file
tl = This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Logon ID = 0xD7FC64F5
Logon Type = 3
Account Name = SVCACCT
Security ID = S-1-5-21-3015042641-2194367929-112691256-2051
hd = Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159648 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Account Domain = MYDOMAIN
-------
tl = This event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.
Logon ID = 0xD7FC600A
Logon Type = 3
Account Name = SVCACCT2
Security ID = S-1-5-21-3015042641-2194367929-112691256-12106
hd = Jan 14 00:00:02 server.host.com MSWinEventLog 5 Security 22159649 Sun Jan 13 23:59:35 2019 4634 Microsoft-Windows-Security-Auditing N/A Audit Success server.host.com 12545 An account was logged off.
Account Domain = MYDOMAIN
-------
使用这种方法，您可以简单地通过名称引用每个字段以进行打印或分析。您可以扩展上面的内容以将第一行中的所有单独字段映射到单独的 tags/values 例如
lineNr==1 {
f["timestamp"] = " " " "
...
}
或使用正则表达式匹配或任何有意义的东西。一旦你[完成了上述操作，在脚本的其余部分分析或打印任何你喜欢的东西就变得非常简单了。