使用 `add_value_provider_argument` 和 apache beam io 函数创建 "RuntimeValueProviderError"
Using `add_value_provider_argument` with apache beam io functions creates "RuntimeValueProviderError"
我正在尝试创建一个将输入参数作为 RuntimeValue
的数据流模板。按照 the docs
中的示例
import re
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
from apache_beam.options.pipeline_options import PipelineOptions
# [START example_wordcount_templated]
class WordcountTemplatedOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
# Use add_value_provider_argument for arguments to be templatable
# Use add_argument as usual for non-templatable arguments
parser.add_value_provider_argument(
'--input', help='Path of the file to read from')
parser.add_argument(
'--output', required=True, help='Output file to write results to.')
pipeline_options = PipelineOptions(['--output', 'some/output_path'])
with beam.Pipeline(options=pipeline_options) as p:
wordcount_options = pipeline_options.view_as(WordcountTemplatedOptions)
lines = p | 'Read' >> ReadFromText(wordcount_options.input)
# [END example_wordcount_templated]
(直接取自 the official snippets)在尝试使用以下命令(已填写具体信息)创建模板时出现以下错误:
python -m examples.mymodule \
--runner DataflowRunner \
--project YOUR_PROJECT_ID \
--staging_location gs://YOUR_BUCKET_NAME/staging \
--temp_location gs://YOUR_BUCKET_NAME/temp \
--template_location gs://YOUR_BUCKET_NAME/templates/YOUR_TEMPLATE_NAME
File "lib/python3.7/site-packages/apache_beam/options/value_provider.py", line 139, in _f
raise error.RuntimeValueProviderError('%s not accessible' % obj)
apache_beam.error.RuntimeValueProviderError: RuntimeValueProvider(option: input, type: str, default_value: None) not
accessible
docs 还指出:
Some I/O connectors contain methods that accept ValueProvider objects.
To determine support for I/O connectors and their methods, see the API
reference documentation for the connector. The following I/O
connectors accept runtime parameters:
File-based IOs: textio, avroio, tfrecordio
我不确定示例代码为何出错。有人可以帮帮我吗?
我正在使用的物有所值:
apache-beam = {extras = ["gcp"], version = "^2.19.0"}
这已在 beam 0.20.0
中修复,自 4/15/2020 发布。
将 beam.io 操作(例如 beam.io.ReadFromText(wordcount_options.input)
与 RuntimeValueProvider 一起使用很简单。如果你遇到我在问题中遇到的相同错误,请尝试升级你的 beam 依赖版本。
- 相关的 JIRA 问题是:BEAM-9218
- 相关 PR 是:beam#10728
我正在尝试创建一个将输入参数作为 RuntimeValue
的数据流模板。按照 the docs
import re
import apache_beam as beam
from apache_beam.io import ReadFromText
from apache_beam.io import WriteToText
from apache_beam.options.pipeline_options import PipelineOptions
# [START example_wordcount_templated]
class WordcountTemplatedOptions(PipelineOptions):
@classmethod
def _add_argparse_args(cls, parser):
# Use add_value_provider_argument for arguments to be templatable
# Use add_argument as usual for non-templatable arguments
parser.add_value_provider_argument(
'--input', help='Path of the file to read from')
parser.add_argument(
'--output', required=True, help='Output file to write results to.')
pipeline_options = PipelineOptions(['--output', 'some/output_path'])
with beam.Pipeline(options=pipeline_options) as p:
wordcount_options = pipeline_options.view_as(WordcountTemplatedOptions)
lines = p | 'Read' >> ReadFromText(wordcount_options.input)
# [END example_wordcount_templated]
(直接取自 the official snippets)在尝试使用以下命令(已填写具体信息)创建模板时出现以下错误:
python -m examples.mymodule \
--runner DataflowRunner \
--project YOUR_PROJECT_ID \
--staging_location gs://YOUR_BUCKET_NAME/staging \
--temp_location gs://YOUR_BUCKET_NAME/temp \
--template_location gs://YOUR_BUCKET_NAME/templates/YOUR_TEMPLATE_NAME
File "lib/python3.7/site-packages/apache_beam/options/value_provider.py", line 139, in _f
raise error.RuntimeValueProviderError('%s not accessible' % obj)
apache_beam.error.RuntimeValueProviderError: RuntimeValueProvider(option: input, type: str, default_value: None) not
accessible
docs 还指出:
Some I/O connectors contain methods that accept ValueProvider objects. To determine support for I/O connectors and their methods, see the API reference documentation for the connector. The following I/O connectors accept runtime parameters:
File-based IOs: textio, avroio, tfrecordio
我不确定示例代码为何出错。有人可以帮帮我吗?
我正在使用的物有所值:
apache-beam = {extras = ["gcp"], version = "^2.19.0"}
这已在 beam 0.20.0
中修复,自 4/15/2020 发布。
将 beam.io 操作(例如 beam.io.ReadFromText(wordcount_options.input)
与 RuntimeValueProvider 一起使用很简单。如果你遇到我在问题中遇到的相同错误,请尝试升级你的 beam 依赖版本。
- 相关的 JIRA 问题是:BEAM-9218
- 相关 PR 是:beam#10728