如何使用 shell 脚本将日志文件转换为 JSON 格式？

Question

我有三行系统日志，我需要将此数据转换为 JSON 以便使用 fluentd 将其转发到 elasticsearch。有没有内部方法可以做到这一点，还是我们必须转换日志然后转发。

示例日志

Feb  3 17:26:36 servername augenrules: failure 1
Feb  3 17:26:38 servername NetworkManager[830]: <info>  [1580768798.0902] manager: rfkill: WiFi enabled by radio killswitch; enabled by state file"}
Feb  3 17:26:44 servername Unit metricbeat.service entered failed state
Feb  3 18:01:44 servername rtprocess[4815]: (RTI rtscantask[6106]@xx)#app#ClsRtdb_query_attr.cpp#931#ClsRtdbAttrTbl::fetchScalarVal(): Failed to fetch scalar value because failed to convert address to number address.
Feb  3 19:26:36 servername augenrules: failure 1
Feb  3 19:45:38 servername NetworkManager[830]: <info>  [1580768798.0902] manager: rfkill: WiFi enabled by radio killswitch; enabled by state file
Feb  3 20:26:44 servername Unit metricbeat.service entered failed state.
Feb  3 21:01:44 servername rtprocess[4815]: (RTI rtscantask[6106]@xx)#app#ClsRtdb_query_attr.cpp#931#ClsRtdbAttrTbl::fetchScalarVal(): Failed to fetch scalar value because failed to convert address to number address.

需要输出

{"date":"Feb  3 17:26:36","server":"servername","error":"augenrules: failure 1"}
{"date":"Feb  3 17:26:38","server":"servername","error":"NetworkManager[830]: <info>  [1580768798.0902] manager: rfkill: WiFi enabled by radio killswitch; enabled by state file"}
{"date":"Feb  3 17:26:44","server":"servername","error":"Unit metricbeat.service entered failed state."}
{"date":"Feb  3 18:01:44","server":"servername","error":"rtprocess[4815]: (RTI rtscantask[6106]@servername)#app#ClsRtdb_query_attr.cpp#931#ClsRtdbAttrTbl::fetchScalarVal(): Failed to fetch scalar value because failed to convert address to number address."}
{"date":"Feb  3 19:26:36","server":"servername","error":"augenrules: failure 1"}
{"date":"Feb  3 19:45:38","server":"servername","error":"NetworkManager[830]: <info>  [1580768798.0902] manager: rfkill: WiFi enabled by radio killswitch; enabled by state fileservername
{"date":"Feb  3 20:26:44","server":"atlswdo087","error":"Unit metricbeat.service entered failed state."}
{"date":"Feb  3 21:01:44","server":"servername","error":"rtprocess[4815]: (RTI rtscantask[6106]@servername)#app#ClsRtdb_query_attr.cpp#931#ClsRtdbAttrTbl::fetchScalarVal(): Failed to fetch scalar value because failed to convert address to number address."}

请帮忙!!!

Answer 1

当然，对于这种情况，您可以使用 logstash and write a GROK matching the message. If you don't want to use logstash, there are ingest pipelines in elastisearch。

请看这篇详细的博文post：https://towardsdatascience.com/from-scratch-to-search-playing-with-your-data-elasticsearch-ingest-pipelines-6d054bf5d866

对于这种情况，您不熟悉 GROK，这里有一个符合您规格的模式：

%{SYSLOGTIMESTAMP:date}%{SPACE}%{IPORHOST:server}%{SPACE}%{GREEDYDATA:error}

编辑：为什么不通过 shell 解决这个问题？因为随着 tgey 的到来，需要不断地处理日志。

Answer 2

This is the Solution I have come up with we can use the same in a while loop to convert the whole log to Json :D 

Feb  3 17:26:36 servername augenrules: failure 1

dt=$(echo $line | awk '{print (" "" " )}')
ser_nm=$(echo $line | awk '{print ()}')
error_msg=$(echo $line | awk '{print ()}')
echo {\"date\":\"$dt\"','"\"server\":\"$ser_nm\",\"error\":\"$error_msg\""}

- Sai Kumar(Bigger the problem bigger the smile when it is resolved)

如何使用 shell 脚本将日志文件转换为 JSON 格式？

How to convert a log file to JSON format using shell script?

linux

elasticsearch

fluentd