BQ 加载：加载期间收到 "unexpected error"，没有额外的输出

Question

我向 Google BigQuery 提交了一个加载作业，它从 google 云存储加载 12 个压缩 (gzip) 表格文件。每个文件压缩约 2 GB。命令 I 运行类似于：

bq load --nosync --skip_leading_rows=1 --source_format=CSV
--max_bad_records=14000 -F "\t" warehouse:some_dataset.2014_lines
gs://bucket/file1.gz,gs://bucket/file2.gz,gs://bucket/file12.gz 
schema.txt

我从我的 BigQuery 加载作业中收到以下错误，但没有解释原因：

Error Reason:internalError. Get more information about this error at Troubleshooting Errors: internalError.

Errors: Unexpected. Please try again.

我确定架构文件的格式正确，因为我已经使用相同的架构但不同的文件集成功加载了文件。

我想知道在什么样的情况下会发生这样的内部错误，我可以通过哪些方式来调试这个问题？

我的 BQ 职位编号：bqjob_r78ca777a8ad4bdd9_0000014e2dc86e0e_1

谢谢！

Answer 1

在某些情况下，您可能会遇到大型 .gz 输入文件，这些文件并不总是以明确的原因报告。对于高度可压缩的文本，尤其（但不限于）会发生这种情况，因此 1 GB 的压缩数据代表异常大量的文本。

压缩 CSV/JSON 的 this page 记录限制为 1 GB。如果这是最新的，我实际上预计您的 2 GB 输入会出现错误。让我检查一下。

您能否将这些文件拆分成更小的部分，然后再试一次？

（Meta：Grace，你是对的，Google says that "Google engineers monitor and answer questions with the tag google-bigquery" on Whosebug. I am a Google engineer, but there are also many knowledgeable people here who are not. Google's docs could perhaps give more explicit guidance: the questions that are most valuable to the Whosebug community are ones that a future person can identify they're seeing this same problem, and preferably that a non-Googler can answer it from public information. It's tough in your case because the error is broad and the cause is unclear. But if you're able to reproduce the problem using an input file that you can make public, more people here will be able to take a crack at the problem. You can also file an issue 对于外面 Google 以外的人确实无法解决的问题。）

BQ 加载：加载期间收到 "unexpected error"，没有额外的输出

BQ loading: Received "unexpected error" during loading with no additional output

google-bigquery