将 Salesforce 批量 api 数据解析为要处理的块

Question

我有下面的代码，我从大量 APi 查询作业中获取结果，该作业的记录数量巨大（60,000 到 80,000）。我正在将其转换为 JSON 并将插入到数据库中。谁能建议处理大量数据的最佳方法以及如何分块处理数据。

request.get( options, function ( error, response, body ) {
        if ( error )
        {

        } else
        {
            csvJson()
                .fromString( response.body )
                .then( ( jsonObj ) => {
                    var a = JSON.stringify( jsonObj );
                } )
        }
    } );

Answer 1

async function* readApi() {
  let page = 1;
  while (page != null) {
    const r = await fetch(`http://target-api.com/stuff?page=${page}`)
    const d = await r.json()
    page = yield d
  }
}

const it = readApi()

it.next() // Init fn with first next call it will get the first page
it.next(2) // Gets the second page, process the data and go to next call
// ..
// ..
it.next(null) // When you are done with getting data call with null

将 Salesforce 批量 api 数据解析为要处理的块

parsing Salesforce bulk api data into chunks to process

javascript

node.js

salesforce-lightning