下载大 CSV 文件并放入 Google Sheet

Question

这是我在自动化方面的一个小项目。我定期检索电子邮件报告 CSV 附件并使用 Google 应用程序脚本将其直接转换为 Google Sheet。但是有报告说太大了，不符合blob限制大小（50mb），会出现执行错误。

因此，下载它并将其存储在 google 驱动器中不是一种选择。

我尝试存储 contentText 并使用我在网上找到的这个 CSVToArray 函数

function CSVToArray( strData, strDelimiter ) {
  // Check to see if the delimiter is defined. If not,
  // then default to COMMA.
  strDelimiter = (strDelimiter || ",");
  // Create a regular expression to parse the CSV values.
  var objPattern = new RegExp(
    (
      // Delimiters.
      "(\" + strDelimiter + "|\r?\n|\r|^)" +

      // Quoted fields.
      "(?:\"([^\"]*(?:\"\"[^\"]*)*)\"|" +

      // Standard fields.
      "([^\"\" + strDelimiter + "\r\n]*))"
    ),
    "gi"
  );

  // Create an array to hold our data. Give the array
  // a default empty first row.
  var arrData = [[]];

  // Create an array to hold our individual pattern
  // matching groups.
  var arrMatches = null;

  // Keep looping over the regular expression matches
  // until we can no longer find a match.
  while (arrMatches = objPattern.exec( strData )){
    // Get the delimiter that was found.
    var strMatchedDelimiter = arrMatches[ 1 ];
    // Check to see if the given delimiter has a length
    // (is not the start of string) and if it matches
    // field delimiter. If id does not, then we know
    // that this delimiter is a row delimiter.
    if (
      strMatchedDelimiter.length &&
      (strMatchedDelimiter != strDelimiter)
    ){

      // Since we have reached a new row of data,
      // add an empty row to our data array.
      arrData.push( [] );

    }
    // Now that we have our delimiter out of the way,
    // let's check to see which kind of value we
    // captured (quoted or unquoted).
    if (arrMatches[ 2 ]){
      // We found a quoted value. When we capture
      // this value, unescape any double quotes.
      var strMatchedValue = arrMatches[ 2 ].replace(
        new RegExp( "\"\"", "g" ),
        "\""
      );
    } else {
      // We found a non-quoted value.
      var strMatchedValue = arrMatches[ 3 ];
    }
    // Now that we have our value string, let's add
    // it to the data array.
    arrData[ arrData.length - 1 ].push( strMatchedValue );
  }
  // Return the parsed data.
  Logger.log(arrData);
  return( arrData );
};

function GetCSVFromLink(link){

  var urlData = UrlFetchApp.fetch(link);
  var stringData = urlData.getContentText(); 
  //
  //All the folder creation etc is here
  //
    var CSVArray = CSVToArray(stringData);   
    var newsheet = ss.insertSheet("NewReport");
    for ( var i =0, lenCsv=CSVArray.length; i<lenCsv;i++)
    {
     newsheet.getRange(i+1,1,1,CSVArray[i].length).setValues(new Array(CSVArray[i]));

    }

最后，我收到了reach Maximum execution time。这个特定的报告有 30k 行，因此即使是 30 分钟的长执行时间也无法完成。但是，这适用于其他较小的 csv 文件。（但是当我可以通过驱动器 API 直接转换为 Google sheet 时我不想这样做）

我还发现，如果我将它从 CSV 转换为 xlsm，它会更小，并且在那里转换会更容易。但问题是我无法让 CSV 文件自动下载到我的云端硬盘，而且我不知道如何使用 App 脚本将 CSV 转换为 xlsm。

是否有任何其他解决方法？或者你们认为还有其他可行的方法吗？

Answer 1

您或许可以通过云端硬盘 API 利用可恢复上传。请参阅 Tanaike's 解决方案：

Resumable Upload for Web Apps using Google Apps Script

下载大 CSV 文件并放入 Google Sheet

Downloading Big CSV Files and putting in Google Sheet

google-apps-script

google-sheets

google-drive-api

csv-import