分布式 Tensorflow 重新加载模型失败

Question

我使用 tensorflow 分布式，store model with codes:

hooks=[tf.train.StopAtStepHook(last_step=1000000)]
with tf.train.MonitoredTrainingSession(master=server.target,
                                               is_chief=is_chief,
                                               checkpoint_dir=self.checkpoint_dir,
                                               hooks=hooks,
                                               save_checkpoint_secs=30,
                                               config=session_conf) as self.sess:

重新加载模型：

checkpoint_dir = 'checkpoints'
checkpoint_file = tf.train.latest_checkpoint(checkpoint_dir)
graph = tf.Graph()
with graph.as_default():
    session_conf = tf.ConfigProto(
        allow_soft_placement=FLAGS.allow_soft_placement,
        log_device_placement=FLAGS.log_device_placement)
    sess = tf.Session(config=session_conf)
    with sess.as_default():
        # Load the saved meta graph and restore variables
        saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
        saver.restore(sess, checkpoint_file)

得到错误：

    saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))


 File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\training\saver.py", line 1686, in import_meta_graph
    **kwargs)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 504, in import_scoped_meta_graph
    producer_op_list=producer_op_list)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\importer.py", line 311, in import_graph_def
    op_def=op_def)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Program Files\Python35\lib\site-packages\tensorflow\python\framework\ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'save/RestoreV2_65': Operation was explicitly assigned to /job:ps/task:0/device:CPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.
     [[Node: save/RestoreV2_65 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:ps/task:0/device:CPU:0"](save/Const, save/RestoreV2_65/tensor_names, save/RestoreV2_65/shape_and_slices)]]

重点是 /job:ps/task:0/device:CPU:0

I find it in meta file:
conv-maxpool-2/W
VariableV2"/job:ps/task:0*
dtype0*

模型保存方式不对？还是重载方式不对？

Answer 1

加载图表时需要清除设备分配，即

tf.train.import_meta_graph('...', clear_devices=True)

分布式 Tensorflow 重新加载模型失败

Distributed Tensorflow reload model failed

python

distributed

tensorflow