NodeJS：如何在从文件解析之前读取和修改缓冲区数据？

Question

在 NodeJS 中，我有一个日志文件，我的日志格式如下：

{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},

这些基本上都是对象，用逗号分隔。我需要做的是读取这个文件的内容并将日志转换成一个对象数组（我可以稍后操作）。

我正在尝试类似的东西：

    let fileLogs = "./data/myfile.log";
    fs.readFile(fileLogs, (err, fileLogsContent) => {
      if (err) {
        console.log("cannot read log file");
        return;
      }

      //I know I need to manipulate the fileLogsContent here, before doing JSON.parse

      let logsContent = { ...JSON.parse(fileLogsContent) };
      //do something here with the array of objects 'logsContent'
    });

由于日志文件中的内容不是可以解析的格式，所以上述JSON.parse失败。我的想法是将日志文件带入以下格式：

[
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}},
{"time":"2021-09-23T11:36:18.076Z","type":"info","message":"some message","data":{"id":123}}
]

这意味着我需要将 [ 作为第一个字符添加到前面，并将最后一个 , 替换为 ]。我不知道我该怎么做，因为 fileLogsContent 实际上是一个缓冲区。那么我如何读取内容并进行我提到的操作，以便稍后能够解析它并将其转换为 array of objects 格式？

Answer 1

您可以简单地将每一行包装在一个字符串中，然后在删除尾随逗号后对其调用 JSON.parse。这是一个例子（注意它仍然需要错误处理 ofc）：

const fs = require('fs');
const readline = require('readline');
const readInterface = readline.createInterface({
    input: fs.createReadStream('./input.txt'),
    output: undefined,
    console: false
});

(async () => {
    const resultArray = await new Promise((resolve, reject) => {
        const chunks = [];
        readInterface.on('line', (line) => {
            line = line.substr(0, line.lastIndexOf(','))
            chunks.push(JSON.parse(`${line}`));
        })

        readInterface.on('close', () => {
            resolve(chunks);
        })
    });
    console.log(resultArray); 
})();

Answer 2

这是我按照@eol 的回答中的说明提出的可行解决方案。

const { once } = require('events');
const { createReadStream } = require('fs');
const { createInterface } = require('readline');

(async function processLineByLine() {
  try {
    const rl = createInterface({
      input: createReadStream('./data/myfile.log'),
      crlfDelay: Infinity
    });
    const chunks = [];

    rl.on('line', (line) => {
      // Process the line.
        chunks.push(JSON.parse(`${line.substr(0, line.lastIndexOf(','))}`));
    });

    await once(rl, 'close');

    console.log('File processed. Content = ', chunks);
  } catch (err) {
    console.log("cannot read log file, err = ", err);
  }
})();

NodeJS：如何在从文件解析之前读取和修改缓冲区数据？

NodeJS: How to read and modify buffer data, before parsing it from file?

node.js

buffer

readfile

streamreader

node-streams