Logstash 的 Serilog HTTP Sink 自定义格式

Question

我在我的 .Net Core 项目中使用 Serilog HTTP sink 登录到 Logstash。在 startup.cs 中，我有以下代码来启用 serilog。

 Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .WriteTo.Http("http://mylogstashhost.com:5000").Enrich.WithProperty("user", "xxx").Enrich.WithProperty("serviceName", "yyy")
        .MinimumLevel.Warning()
        .CreateLogger();

此代码将日志发送到给定的 http 地址。我可以在 fiddler 上看到以下 json 被发布到 logstash 和 logstash returns "ok" 消息。

{"events":[{"Timestamp":"2018-10-19T18:16:27.6561159+01:00","Level":"Warning","MessageTemplate":"abc","RenderedMessage":"abc","user":"xxx","serviceName":"yyy","Properties":{"ActionId":"b313b8ed-0baf-4d75-a6e2-f0dbcb941f67","ActionName":"MyProject.Controllers.HomeController.Index","RequestId":"0HLHLQMV1EBCJ:00000003","RequestPath":"/"}}]}

但是我在Kibana上查看的时候，看不到这个日志。我试图弄清楚是什么原因造成的，我意识到如果我按以下格式发送 json 我可以看到日志。

{"Timestamp":"2018-10-19T18:16:27.6561159+01:00","Level":"Warning","MessageTemplate":"abc","RenderedMessage":"abc","user":"xxx","serviceName":"yyy","Properties":{"ActionId":"b313b8ed-0baf-4d75-a6e2-f0dbcb941f67","ActionName":"MyProject.Controllers.HomeController.Index" ,"RequestId":"0HLHLQMV1EBCJ:00000003","RequestPath":"/"}}

所以 Logstash 不希望事件出现在 Events{} 中，它还希望 "user" 和 "ServiceName" 标签不在 "Properties" 中。有没有办法像这样格式化我的 Json？

Answer 1

好的，经过一些研究和帮助，基本上要实现自定义格式，应该实现 ITextFormatter、BatchFormatter 等接口

我可以通过稍微修改 ArrayBatchFormatter 来实现我需要的格式：

public class MyFormat : BatchFormatter
{
    /// <summary>
    /// Initializes a new instance of the <see cref="ArrayBatchFormatter"/> class.
    /// </summary>
    /// <param name="eventBodyLimitBytes">
    /// The maximum size, in bytes, that the JSON representation of an event may take before it
    /// is dropped rather than being sent to the server. Specify null for no limit. Default
    /// value is 256 KB.
    /// </param>
    public MyFormat(long? eventBodyLimitBytes = 256 * 1024): base(eventBodyLimitBytes)
    {

    }

    /// <summary>
    /// Format the log events into a payload.
    /// </summary>
    /// <param name="logEvents">
    /// The events to format.
    /// </param>
    /// <param name="output">
    /// The payload to send over the network.
    /// </param>
    public override void Format(IEnumerable<string> logEvents, TextWriter output)
    {
        if (logEvents == null) throw new ArgumentNullException(nameof(logEvents));
        if (output == null) throw new ArgumentNullException(nameof(output));

        // Abort if sequence of log events is empty
        if (!logEvents.Any())
        {
            return;
        }

        output.Write("[");

        var delimStart = string.Empty;

        foreach (var logEvent in logEvents)
        {
            if (string.IsNullOrWhiteSpace(logEvent))
            {
                continue;
            }
            int index = logEvent.IndexOf("{");

            string adjustedString = "{\"user\":\"xxx\",\"serviceName\" : \"yyy\"," + logEvent.Substring(1);
            if (CheckEventBodySize(adjustedString))
            {
                output.Write(delimStart);
                output.Write(adjustedString);
                delimStart = ",";
            }
        }

        output.Write("]");
    }
}

Answer 2

我想用这个变体来扩展@nooaa 的答案。我建议使用 Newtonsoft.Json.Linq，而不是操纵字符串来添加新对象。这样您就可以附加、添加或删除对象本身的现有属性。

此外，您可以组合事件的所有输出并在结束时执行一次 output.write，而不是在每个事件之后执行 output.write（有点性能）

public override void Format(IEnumerable<string> logEvents, TextWriter output)
{
    if (logEvents == null) throw new ArgumentNullException(nameof(logEvents));
    if (output == null) throw new ArgumentNullException(nameof(output));

    // Abort if sequence of log events is empty
    if (!logEvents.Any())
    {
        return;
    }
    
    List<object> updatedEvents = new List<object>();
    foreach (string logEvent in logEvents)
    {
        if (string.IsNullOrWhiteSpace(logEvent))
        {
            continue;
        }

        // Parse the log event
        var obj = JObject.Parse(logEvent);

        // Add New entries 
        obj["@source_host"] = obj["fields"]["MachineName"].Value<string>().ToLower();

        // Remove any entries you are not interested in
        ((JObject)obj["fields"]).Remove("MachineName");
                
        // Default tags for any log that goes out of your app.
        obj["@tags"] = new JArray() { "appName", "api" };

        // Additional tags from end points (custom based on routes)
        if (obj["fields"]["tags"] != null) 
        {
            ((JArray)obj["@tags"]).Merge((JArray)obj["fields"]["tags"]);
            ((JObject)obj["fields"]).Remove("tags");
        }
       
        updatedEvents.Add(obj);
    }

    output.Write(JsonConvert.SerializeObject(updatedEvents));
}

Logstash 的 Serilog HTTP Sink 自定义格式

Serilog HTTP Sink custom formatting for Logstash

format

json

logstash

serilog

asp.net-core