apache_beam.runners.dataflow_runner.DataflowRuntimeException：数据流管道失败：

Question

我在 Cloud Shell 中设置了一个 Google Cloud 项目，并尝试运行本教程脚本 https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/flowers/sample.sh

运行进入这个错误：

***@***:~/git/cloudml-samples/flowers$ ./sample.sh 
Your active configuration is: [cloudshell-4691]


Using job id:  flowers_***_20170113_162148

python trainer/preprocess.py \
  --input_dict "$DICT_FILE" \
  --input_path "gs://cloud-ml-data/img/flower_photos/eval_set.csv" \
  --output_path "${GCS_PATH}/preproc/eval" \
  --cloud
WARNING:root:Using fallback coder for typehint: Any.
WARNING:root:Using fallback coder for typehint: Any.
WARNING:root:Using fallback coder for typehint: Any.
DEPRECATION: pip install --download has been deprecated and will be removed in the future. Pip now has a download command that should be used instead.
Collecting google-cloud-dataflow==0.4.4
  Using cached google-cloud-dataflow-0.4.4.zip
  Saved /tmp/tmpSoHiTi/google-cloud-dataflow-0.4.4.zip
Successfully downloaded google-cloud-dataflow
# Takes about 30 mins to preprocess everything.  We serialize the two
Traceback (most recent call last):
  File "trainer/preprocess.py", line 436, in <module>
    main(sys.argv[1:])
  File "trainer/preprocess.py", line 432, in main
    run(arg_dict)
  File "trainer/preprocess.py", line 353, in run
    p.run()
  File "/home/slalomconsultingsf/.local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 159, in run
    return self.runner.run(self)
  File "/home/slalomconsultingsf/.local/lib/python2.7/site-packages/apache_beam/runners/dataflow_runner.py", line 195, in run
    % getattr(self, 'last_error_msg', None), self.result)
apache_beam.runners.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed:
(b85b0a598a3565cb): Workflow failed.

我无法从 the Error Log of GoogleCloud Dataflow

中找到我哪里做错的线索

感谢任何答案和故障排除帮助。

Answer 1

启用数据流API。在 Pantheon 顶部搜索框中键入 "dataflow api" 将带您进入 window，您可以在其中单击 "Enable API"。

我想这会为您解决问题。我禁用了我的数据流 API 并得到了与你相同的错误，然后当它是 re-enabled 时问题又消失了。

apache_beam.runners.dataflow_runner.DataflowRuntimeException：数据流管道失败：

apache_beam.runners.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed:

google-cloud-dataflow

google-cloud-ml

apache-beam