是否可以将带有 Snappy 压缩的 Avro 文件加载到 BigQuery 中?

Is it possible to load Avro files with Snappy compression into BigQuery?

我知道 BigQuery 支持 Avro 文件上传,我已成功将 Avro 文件加载到 BigQuery。

使用下面的命令,

java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro

我生成了一个带有 Snappy 压缩的 Avro 文件并尝试加载到 BigQuery 但加载作业失败并出现以下错误,

Errors:
file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid)

是否可以将采用 Snappy 压缩的 Avro 文件加载到 BigQuery 中?

BigQuery 仅支持用于 Avro 数据块压缩的 DEFLATE 和 Snappy 算法,来自文档 (https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro)

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.

现在 BigQuery 支持 Snappy。参见:https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro

Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.