Bash 如何有效地操作 grep -Poz 多行输出？

Question

这是我在 Whosebug 上的第一个 post。 \0/ 我希望它不会太长。我正在编写一个 BASH 脚本来定期读取、过滤和输出来自数千个日志文件的数据。性能很重要，所以我主要使用 grep 而不是 awk 或 sed。

grep -Poz 完全按照我的意愿使用与进一步处理相关的模式捕获（多行）数据，但我一直坚持将数据处理为 XML 文件或用于进一步分析的 SQLite3 批查询。

#!/bin/bash
# Regex:
# (?s) multiline search
# Capturegroup 1 = date
# Capturegroup 2 = time
# Capturegroup 3 = error type (ERROR, WARN or DEBUG)
# Capturegroup 4 = error details
# Positive lookahed, until new line (windows/linux) starts with date, OR (if it's the last line matching the pattern, till the end of the last line.
#
REGEX_MULTILINE="(?s)([0-9]{4}-[0-9]{2}-[0-9]{2})[[:space:]]([0-9]{2}:[0-9]{2}:[0-9]{2}[,|.][0-9]{3})[[:space:]]+(ERROR|WARN|DEBUG)(.*?)(?=(?:\r\n|[\r\n])[0-9]{4}-[0-9]{2}-[0-9]{2}|\z)"
LOGFILE="test.log"

# write to logfile gives exactly the info I want
write_log(){
    echo -n $(grep -Pzo $REGEX_MULTILINE $LOGFILE) > output_grep1.txt
}

# I'm stuck in this part to generate, for example, an XML-file
write_xml(){
    local LOGDATE=""
    local LOGTIME=""
    local LOGTYPE=""
    local LOGINFO=""
    while IFS= read -r LINE ; do
    #For testing purposes, to see if brackets contain the full string, 
    #or a line of that string
    printf '%s\n' "[$LINE]"
    #processing logic here. Didn't get this far yet
    while [[ $LINE =~ $REGEX_MULTILINE ]] ; do
        # regex capturegoups
        LOGDATE=${BASH_REMATCH[1]}
        LOGTIME=${BASH_REMATCH[2]}
        LOGTYPE=${BASH_REMATCH[3]}
        LOGINFO=${BASH_REMATCH[4]}
        # send vars to function for output
        # write_xml_function $LOGDATE $LOGTIME $LOGTYPE $LOGINFO
        # for testing purposes
        echo -e "log entry:\n\t 1: $LOGDATE \n\t 2: $LOGTIME \n\t 3: $LOGTYPE \n\t 4: $LOGINFO \n" 
        break
    done
done < <(grep -Pzo $REGEX_MULTILINE $LOGFILE)
}

日志文件可能如下所示：

2017-01-01 11:09:42,439 INFO  server.service.function.property.PropertyService - Props (re)loaded.
2017-01-01 11:15:46,155 DEBUG server.service.ApiController - api/start called! params:
${params}
2017-01-01 13:01:29,675 ERROR server.service.util.base.FtpClient - Error retrieving file. Directory does not exist.
2017-01-01 13:15:12,803 DEBUG server.service.ApiController - api/start called! params:
${params}
2017-01-01 13:15:13,932 INFO server.service.ControllerService - Filter:server.service.model.Filters
2017-01-01 15:36:04,914 INFO server.service.ControllerService - Filter:server.service.model.Filters
2017-01-01 15:55:50,279 ERROR server.service.WebClient - server API failed: [(someError.java:12345)]
{"someId":"etc","otherId":123,"token":{}}
2017-01-01 15:55:50,366 ERROR server.service.controller.Search - Server error for [/service/search/load]: java.lang.NullPointerException stack[etc]
java.lang.NullPointerException
    at server.common.stack(SomeApi.java:123)
    at server.service.trace(SomeService.java:456)
    at java.lang.Thread.run(Thread.java:789)
    etc.
    etc.
2017-01-01 16:17:55,175 DEBUG server.config.app - 

STARTING...


2017-01-01 16:18:00,040 INFO  server.common.service.base.property - Props (re)loaded.
2017-01-01 17:44:43,959 DEBUG server.service.controller - api/start called! params:
${params}

我期望读取 grep 多行字符串的结果是这样的：

[2017-01-01 13:15:13,932 INFO server.service.ControllerService - Filter:server.service.model.Filters]
[2017-01-01 15:36:04,914 INFO server.service.ControllerService - Filter:server.service.model.Filters]
[2017-01-01 15:55:50,279 ERROR server.service.WebClient - server API failed: [(someError.java:12345)]
{"someId":"etc","otherId":123,"token":{}}]
[2017-01-01 15:55:50,366 ERROR server.service.controller.Search - Server error for [/service/search/load]: java.lang.NullPointerException stack[etc]
java.lang.NullPointerException
    at server.common.stack(SomeApi.java:123)
    at server.service.trace(SomeService.java:456)
    at java.lang.Thread.run(Thread.java:789)
    etc.
    etc.]

相反，我得到了这个：

[2017-01-01 13:15:13,932 INFO server.service.ControllerService - Filter:server.service.model.Filters]
[2017-01-01 15:36:04,914 INFO server.service.ControllerService - Filter:server.service.model.Filters]
[2017-01-01 15:55:50,279 ERROR server.service.WebClient - server API failed: [(someError.java:12345)]
{"someId":"etc","otherId":123,"token":{}}]
[2017-01-01 15:55:50,366 ERROR server.service.controller.Search - Server error for [/service/search/load]: java.lang.NullPointerException stack[etc]]
[java.lang.NullPointerException]
[   at server.common.stack(SomeApi.java:123)]
[   at server.service.trace(SomeService.java:456)]
[   at java.lang.Thread.run(Thread.java:789)]
[   etc.]
[   etc.]

我忽略了什么？可以这样做吗？

Answer 1

问题出在您的 read 命令上。默认情况下，read 将读取到换行符，但您正在尝试处理 null-separated 个字符串。

您应该可以使用

while IFS= read -r -d '' LINE ; do

Bash 如何有效地操作 grep -Poz 多行输出？

Bash How to efficiently manipulate a grep -Poz multiline output?

regex

bash

grep

pcre

multiline