编写一个 Bash shell 脚本来读取 mongodb 日志文件并将其内容写入另一个文本文件

Question

我正在尝试读取位于 /var/log/mongodb 的 MongoDB 日志文件，其内容如下：

2019-11-04T05:04:00.390-0800 I COMMAND [conn38649] command loldb.$cmd command: update { update: "SUBSCRIPTION", ordered: true, writeConcern: { w: 1 }, $db: "loldb" } numYields:0 reslen:295 locks:{ Global: { acquireCount: { r: 460, w: 460 } }, Database: { acquireCount: { w: 460 } }, Collection: { acquireCount: { w: 459 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_query 568ms

2019-11-04T05:04:00.396-0800 I COMMAND [conn38657] command loldb.SUBSCRIPTION command: find { find: "SUBSCRIPTION", filter: { customerID: 75824180, policeDepartmentID: 1 }, projection: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "loldb" } planSummary: COLLSCAN keysExamined:0 docsExamined:69998 cursorExhausted:1 numYields:550 nreturned:1 reslen:430 locks:{ Global: { acquireCount: { r: 1102 } }, Database: { acquireCount: { r: 551 } }, Collection: { acquireCount: { r: 551 } } } protocol:op_query 424ms

2019-11-04T05:04:00.402-0800 I COMMAND [conn38735] command loldb.SUBSCRIPTION command: find { find: "SUBSCRIPTION", filter: { customerID: 75824164 }, projection: {}, $readPreference: { mode: "secondaryPreferred" }, $db: "loldb" } planSummary: COLLSCAN keysExamined:0 docsExamined:58142 cursorExhausted:1 numYields:456 nreturned:1 reslen:417 locks:{ Global: { acquireCount: { r: 914 } }, Database: { acquireCount: { r: 457 } }, Collection: { acquireCount: { r: 457 } } } protocol:op_query 374ms

每个块引用都是一个单行条目

文件内容每秒更新我需要读取文件，如果查询时间 protocol:op_query 385ms 超过 300 毫秒我需要将整个 log/line 保存到另一个文本文件 slow_queries.text.

我正在阅读的文件是.log文件，但是内容好像是JSON格式（如有错误请指正），前面有时间戳和命令类型，有没有有效的如何读取这种格式的数据？我只是逐字逐句地阅读。

此外，我该怎么做才能让对 .log 文件所做的更改每次都自动读取，而无需运行脚本？

Answer 1

我刚刚在我的本地机器上试过这个，可能需要为你的用例做一些工作。但我添加了一些评论，所以也许这会对你有所帮助：

编辑：我添加了时间戳检查，您必须根据需要配置它

#!/bin/bash

# continously read from the file and pipe it into the while loop
tail -F "test.log" | \
while read LINE
do
    # get timestamp from LINE and get time in seconds
    timeinseconds="$(grep -P "^\S*" | date -d - +%s)"
    # get current timestamp before 5 minutes
    timebeforefivemin="$(date -d '-5 minutes' +%s)"
    # only log if timestamp of line is smaller to time before 5 min
    if [[ $(expr $timeinseconds - $timebeforefivemin) -lt 0 ]];
        then
        # get the time of the query from the line
        querytime="$(echo "$LINE" | grep -oP '\d+ms' | grep -oP '\d+')"
        #if the grep was successful and the query time is greater than 300
        if [ $? = 0 ] && [ "$querytime" -gt 300 ]
        then
            # echo the line into the slow_queries file -> change it to the path you want
            echo "$LINE" >> slow_queries.txt
        fi
    fi
done

编写一个 Bash shell 脚本来读取 mongodb 日志文件并将其内容写入另一个文本文件

Write a Bash shell script to read mongodb log file and write it's content to another text file

bash

shell

cron

mongodb

cron-task