修改 raw_event 以使用 NXLog 发送自定义日志

Question

我想修改IIS日志以便进一步传输到目的地。现在我正在使用 xm_csv 模块解析 IIS 日志，就像在模板中一样。 UndefValue 已禁用，不会为空。

如何与来自 w3c_parser 的解析数据进行交互？

比如我想组合成一个变量 $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';这样的值，但我得到一个错误。当我尝试写入从 w3c_parser 到 $raw_event 的字段时，我也遇到错误。添加任何其他数据都没有错误。

例如$raw_event = $c-ip -- 错误

$raw_event = $EventTime + ' ' + $http_host -- 没有错误

示例错误、日志和配置文件如下

2022-03-23 16:49:56 ERROR Couldn't parse Exec block at C:\Program Files\nxlog\conf\nxlog.conf:59; couldn't parse statement at line 71, character 32 in C:\Program Files\nxlog\conf\nxlog.conf; syntax error, unexpected +, expecting (

2022-03-23 16:49:56 ERROR module 'iis_w3c' has configuration errors, not adding to route 'uds_to_file' at C:\Program Files\nxlog\conf\nxlog.conf:84

2022-03-23 16:49:56 ERROR route uds_to_file is not functional without input modules, ignored at C:\Program Files\nxlog\conf\nxlog.conf:84

2022-03-23 16:49:56 WARNING no routes defined!

2022-03-23 16:49:56 WARNING not starting unused module iis_w3c

2022-03-23 16:49:56 WARNING not starting unused module file

2022-03-23 16:49:56 INFO nxlog-ce-3.0.2272 started

当前日志格式

date time s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken

2022-03-23 08:00:01 HOST.DOMAIN 99.XX.XX.4 GET /AnalyticsService - 443

XX.XX.XX.XXX HTTP/1.1 Zabbix - - site.host.domain 200 3918 144 4

要求的日志格式

$http_host $remote_addr $remote_user [$time_local] UNIX-TIME-$msec "$request" $status "$sent_http_content_type" $body_bytes_sent "$http_referer" "$http_user_agent" "$http_cookie" $request_time "$upstream_addr" NGINX-CACHE-$upstream_cache_status "$request_id" "$request_body"

host.domain 99.99.99.249 - [11/Mar/2022:20:09:56+0300] UNIX-TIME-1647018596.031 "GET /api/company.php?id=853747 HTTP/1.1" 200 "text/xml; charset=UTF-8" 1455 "-" "-" "20b6b325ea192383cb1244412247c5ea=3002538ef353c9daab4f742176a840; etpsid=f488b343a23d1a4a2332e089a0" 0.059 0.059 "10.10.10.111:80" NGINX-CACHE-- "d0b5ac12cf82671067aa5e6c5c" "-"

Panic Soft
#NoFreeOnExit TRUE

define ROOT C:\Program Files\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf\nxlog.d
define LOGDIR %ROOT%\data

define LOGFILE %LOGDIR%\nxlog.log
LogFile %LOGFILE%

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data

<Extension _syslog>
Module xm_syslog
</Extension>

<Extension fileop>
Module xm_fileop
</Extension>

<Extension _charconv>
Module xm_charconv
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
</Extension>

<Extension _exec>
Module xm_exec
</Extension>

#Create the parse rule for IIS logs. You can copy these from the header of the IIS log file.
<Extension w3c_parser>
Module xm_csv
Fields $date, $time, $s-computername, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $cs-version, $cs(User-Agent), $cs(Cookie), $cs(Referer), $cs-host, $sc-status, $sc-bytes, $cs-bytes, $time-taken
FieldTypes string, string, string, string, string, string, string, integer, string, string, string, string, string, string, string, integer, integer, integer, integer
Delimiter ' '
EscapeChar '"'
QuoteChar '"'
EscapeControl FALSE
# UndefValue -
</Extension>

<Extension w3c_out>
Module xm_csv
Fields $http_host, $c-ip, $cs-username, $EventTime1, $sc-status, $Unix
FieldTypes string, string, string, string, string, string
Delimiter ' '
# UndefValue -
QuoteMethod None
</Extension>

<Input iis_w3c>
Module im_file
File 'C:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log'
SavePos TRUE
<Exec>
if $raw_event =~ /^#/ drop();
else
{
w3c_parser->parse_csv();
$EventTime = parsedate($date + " " + $time);
$EventTime = $EventTime + (3 * 3600);
$EventTime1 = strftime($EventTime, '[%d/%b/%Y:%H:%M:%S]');
# $EventTime1 = '$EventTime1' + ' +0003]';
$Unix = integer($EventTime);
$Unix = 'UNIX-TIME-' + $Unix;
$http_host = "site.host.domain";
# $request = '"' + $cs-method + ' ' + $cs-uri-stem + ' ' + $cs-version + '"';
# $request = $cs-method;
w3c_out->to_csv();
}
</Exec>
</Input>

<Output file>
Module om_file
File 'C:\inetpub\logs\LogFiles\Parser\w3c.txt'
</Output>

<Route uds_to_file>
Path iis_w3c => file
</Route>

Answer 1

让我们从 conf 文件中的 NXLog 语言开始。不允许使用显式格式的破折号 - 您可以检查： https://nxlog.co/docs/nxlog-ce/nxlog-reference-manual.html#lang_fields

因此，需要使用花括号才能达到目标({})。如果我对你的问题的理解正确，这可能有助于解决你的大部分问题。

修改 raw_event 以使用 NXLog 发送自定义日志

Modify raw_event to send custom logs using NXLog

parsing

iis

nxlog