写入 tensorflow 转换元数据时，管道将在 GCP 上失败
Pipeline will fail on GCP when writing tensorflow transform metadata
我希望这里有人可以提供帮助。我一直在疯狂地搜索这个错误，但没有找到任何东西。
我的管道在本地执行时运行良好，但在 GCP 上执行时失败。以下是我收到的错误消息。
Workflow failed. Causes: S03:Write transform
fn/WriteMetadata/ResolveBeamFutures/CreateSingleton/Read+Write
transform fn/WriteMetadata/ResolveBeamFutures/ResolveFutures/Do+Write
transform fn/WriteMetadata/WriteMetadata failed., A work item was
attempted 4 times without success. Each time the worker eventually
lost contact with the service. The work item was attempted on:
Traceback (most recent call last): File "preprocess.py", line 491,
in
main() File "preprocess.py", line 487, in main
transform_data(args,pipeline_options,runner) File "preprocess.py", line 451, in transform_data
eval_data |= 'Identity eval' >> beam.ParDo(Identity()) File "/Library/Python/2.7/site-packages/apache_beam/pipeline.py", line 335,
in exit
self.run().wait_until_finish() File "/Library/Python/2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 897, in wait_until_finish
(self.state, getattr(self._runner, 'last_error_msg', None)), self) apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:
Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent
call last): File
"/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 582, in do_work
work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 166, in execute
op.start() File "apache_beam/runners/worker/operations.py", line 294, in apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:10607)
def start(self): File "apache_beam/runners/worker/operations.py", line 295, in
apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:10501)
with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 300, in
apache_beam.runners.worker.operations.DoOperation.start
(apache_beam/runners/worker/operations.c:9702)
pickler.loads(self.spec.serialized_fn)) File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py",
line 225, in loads
return dill.loads(s) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in
loads
return load(file) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in
load
obj = pik.load() File "/usr/lib/python2.7/pickle.py", line 858, in load
dispatchkey File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
obj = cls.new(cls, *args) TypeError: new() takes exactly 4 arguments (1 given)
有什么想法吗？？
谢谢，
佩德罗
如果管道在本地工作但在 GCP 上失败，则可能是您 运行 版本不匹配。
什么 TF，tf.Transform，你是 运行 本地和 GCP 上的 beam 版本？
我希望这里有人可以提供帮助。我一直在疯狂地搜索这个错误，但没有找到任何东西。
我的管道在本地执行时运行良好，但在 GCP 上执行时失败。以下是我收到的错误消息。
Workflow failed. Causes: S03:Write transform fn/WriteMetadata/ResolveBeamFutures/CreateSingleton/Read+Write transform fn/WriteMetadata/ResolveBeamFutures/ResolveFutures/Do+Write transform fn/WriteMetadata/WriteMetadata failed., A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on:
Traceback (most recent call last): File "preprocess.py", line 491, in main() File "preprocess.py", line 487, in main transform_data(args,pipeline_options,runner) File "preprocess.py", line 451, in transform_data eval_data |= 'Identity eval' >> beam.ParDo(Identity()) File "/Library/Python/2.7/site-packages/apache_beam/pipeline.py", line 335, in exit self.run().wait_until_finish() File "/Library/Python/2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 897, in wait_until_finish (self.state, getattr(self._runner, 'last_error_msg', None)), self) apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 582, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 166, in execute op.start() File "apache_beam/runners/worker/operations.py", line 294, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:10607) def start(self): File "apache_beam/runners/worker/operations.py", line 295, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:10501) with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 300, in apache_beam.runners.worker.operations.DoOperation.start (apache_beam/runners/worker/operations.c:9702) pickler.loads(self.spec.serialized_fn)) File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line 225, in loads return dill.loads(s) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 277, in loads return load(file) File "/usr/local/lib/python2.7/dist-packages/dill/dill.py", line 266, in load obj = pik.load() File "/usr/lib/python2.7/pickle.py", line 858, in load dispatchkey File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj obj = cls.new(cls, *args) TypeError: new() takes exactly 4 arguments (1 given)
有什么想法吗？？
谢谢，
佩德罗
如果管道在本地工作但在 GCP 上失败，则可能是您 运行 版本不匹配。
什么 TF，tf.Transform，你是 运行 本地和 GCP 上的 beam 版本？