BigQuery - 使用换行符加载 CSV 失败并显示 Node.js，但适用于 gsutil

Question

我正在尝试使用 Cloud Function 将 CSV 从 Cloud Storage 加载到 Big Query。该文件在某些字符串中有换行符。

当我从云 Shell 加载时，它加载完美，并且在 table 中找到带有换行符的字符串。

bq --location=EU load \
--replace \
--allow_quoted_newlines \
--skip_leading_rows=1 \
--source_format=CSV \
my_gcp_project:my_dataset.my_table \
gs://9604/data/data.csv \
my_schema.json

在我的 Cloud Function 中出现错误，

Error while reading data, error message: Error detected while parsing row starting at position: 2624400. Error: Missing close double quote (") character.

将 Cloud Function 与 Node.js 10 一起使用，对我来说设置相同；

const datasetId = 'my_dataset';
    const tableId = 'my_table';
    const metadata = {
      sourceFormat: 'CSV',
      skipLeadingRows: 1,
      allowQuotedNewlines: True,
      schema: {
        fields: [{"name": "gdb", "type": "STRING"},
  {"name": "venueid", "type": "STRING"},
  {"mode": "NULLABLE", "name": "buildingid", "type": "STRING"},
  {"mode": "NULLABLE", "name": "floorexists", "type": "STRING"},
  {"mode": "NULLABLE", "name": "roomid", "type": "STRING"},
  {"mode": "NULLABLE", "name": "displayname", "type": "STRING"},
  {"mode": "NULLABLE", "name": "zlevel", "type": "INTEGER"},
  {"mode": "NULLABLE", "name": "class", "type": "STRING"},
  {"mode": "NULLABLE", "name": "accesslevel", "type": "STRING"},
  {"mode": "NULLABLE", "name": "created_date", "type": "DATETIME"},
  {"mode": "NULLABLE", "name": "year", "type": "STRING"},
  {"mode": "NULLABLE", "name": "quarter", "type": "STRING"},
  {"mode": "NULLABLE", "name": "yearquarter", "type": "STRING"},
  {"mode": "NULLABLE", "name": "month", "type": "STRING"},
  {"mode": "NULLABLE", "name": "area_sqm", "type": "FLOAT"},
  {"mode": "NULLABLE", "name": "area_sqft", "type": "FLOAT"}],
      },
      // Set the write disposition to overwrite existing table data.
      writeDisposition: 'WRITE_TRUNCATE',
      location: 'EU',
    };

我错过了什么？感谢您的输入！

Answer 1

发现问题。对于架构，我有：

{'name': 'gdb', 'type' : 'STRING'},
{'name': 'venueid', 'type' : 'STRING'},

删除 '' 解决了这个问题。因此，使用 GSUTIL 没问题，但在 JS 中就没那么好了。工作模式：

{name: 'gdb', type: 'STRING'},
{name: 'venueid', type: 'STRING'}

BigQuery - 使用换行符加载 CSV 失败并显示 Node.js，但适用于 gsutil

BigQuery - loading CSV with newlines fail with Node.js, but works with gsutil

csv

google-bigquery