数据流 IO 到 BigTable [2.9.0]

Question

我找到了这个 Bigtable with Dataflow 示例 https://github.com/GoogleCloudPlatform/cloud-bigtable-examples/blob/master/java/dataflow-connector-examples/src/main/java/com/google/cloud/bigtable/dataflow/example/HelloWorldWrite.java

但是；它使用

beam-runners-google-cloud-dataflow-java 2.4.0

并且在 2.9.0 中 org.apache.beam.runners.dataflow.options.DataflowPipelineOptions 不再存在。

是否有从 Dataflow 写入 BigTable 的最新示例？

我发现：https://beam.apache.org/releases/javadoc/2.0.0/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.html - 在正确的轨道上吗？

Answer 1

我使用了 BigTable 连接器 example you link to, following the instructions here, with Dataflow Java SDK 2.9.0 and it works fine. The only extra step needed is to change the SDK in the pom.xml file (line):

将 <beam.version>2.4.0</beam.version> 替换为 <beam.version>2.9.0</beam.version>。

Dataflow 作业将开始（您将在标准输出中看到 Dataflow SDK version: 2.9.0）。一旦成功，您可以在 HBase shell 中验证是否写入了正确的行：

hbase(main):001:0> scan 'Dataflow_test'
ROW                                                                              COLUMN+CELL
 Hello                                                                           column=cf:qualifier, timestamp=1548151071821, value=value_21.60451762361535
 World                                                                           column=cf:qualifier, timestamp=1548151064955, value=value_21.60451762361535
2 row(s) in 1.4230 seconds

数据流 IO 到 BigTable [2.9.0]

Dataflow IO to BigTable [2.9.0]

google-cloud-platform

google-cloud-dataflow

google-cloud-bigtable

apache-beam-io