数据流：从 Pubsub RuntimeException 导出到 Bigquery
Dataflow: Export to Bigquery from Pubsub RuntimeException
我正在使用 pubsub 中的 "export to bigquery" 功能通过数据流将常规 JSON 从 pubsub 传递到 bigquery。
但是它只工作了一秒钟，这意味着一些条目正确地通过了 bigquery。但是现在我在数据流日志中收到错误
java.lang.RuntimeException: java.io.IOException: Insert failed:
[{"errors":[{"debugInfo":"","location":"_comments","message":"no such
field.","reason":"invalid"}],"index":0}]
org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:131)
org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:97)
Caused by: java.io.IOException: Insert failed:
[{"errors":[{"debugInfo":"","location":"_comments","message":"no such
field.","reason":"invalid"}],"index":0}]
... MANY MANY LINES...
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:811)
org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:127)
org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:97)
org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn$DoFnInvoker.invokeFinishBundle(Unknown
Source)
org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:187)
com.google.cloud.dataflow.worker.SimpleParDoFn.finishBundle(SimpleParDoFn.java:407)
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:60)
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1069)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access00(StreamingDataflowWorker.java:133)
com.google.cloud.dataflow.worker.StreamingDataflowWorker.run(StreamingDataflowWorker.java:841)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Pub/Sub 中的字段与 Big Query 中的字段似乎不匹配。
检查你的两边字段名称是否相同。您可以在 here
中查看有关 Dataflow 模板的更多信息
我正在使用 pubsub 中的 "export to bigquery" 功能通过数据流将常规 JSON 从 pubsub 传递到 bigquery。
但是它只工作了一秒钟，这意味着一些条目正确地通过了 bigquery。但是现在我在数据流日志中收到错误
java.lang.RuntimeException: java.io.IOException: Insert failed: [{"errors":[{"debugInfo":"","location":"_comments","message":"no such field.","reason":"invalid"}],"index":0}] org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:131) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:97) Caused by: java.io.IOException: Insert failed: [{"errors":[{"debugInfo":"","location":"_comments","message":"no such field.","reason":"invalid"}],"index":0}]
... MANY MANY LINES...
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:811) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:127) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:97) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source) org.apache.beam.runners.core.SimpleDoFnRunner.finishBundle(SimpleDoFnRunner.java:187) com.google.cloud.dataflow.worker.SimpleParDoFn.finishBundle(SimpleParDoFn.java:407) com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.finish(ParDoOperation.java:60) com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:76) com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1069) com.google.cloud.dataflow.worker.StreamingDataflowWorker.access00(StreamingDataflowWorker.java:133) com.google.cloud.dataflow.worker.StreamingDataflowWorker.run(StreamingDataflowWorker.java:841) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745)
Pub/Sub 中的字段与 Big Query 中的字段似乎不匹配。
检查你的两边字段名称是否相同。您可以在 here
中查看有关 Dataflow 模板的更多信息