由于保存模型导致训练崩溃："tensorflow.GraphDef was modified concurrently during serialization"

Question

我目前正在尝试训练模型，我的输入管道是根据此答案构建的 . I want to save my model after each epochs. But after training for some epochs the training crash. I have read that it is because it adds the input as a constant tensor to the graph. There are suggested solutions here 以使用 tf.placeholder 来解决问题。不幸的是，它并没有为我解决问题。输入管道如下所示

....
filenames = [P_1]
dataset = tf.data.TFRecordDataset(filenames)
def _parse_function(example_proto):
       keys_to_features = { 'data':tf.VarLenFeature(tf.float32)},
       parsed_features = tf.parse_single_example(example_proto,  keys_to_features)
       return tf.sparse_tensor_to_dense(parsed_features['data'
# Parse the record into tensors.
dataset = dataset.map(_parse_function)
# Shuffle the dataset
dataset = dataset.shuffle(buffer_size=1000)
# Repeat the input indefinitly 
dataset = dataset.repeat()      
# Generate batches     
dataset = dataset.batch(Batch_size) 
# Create a one-shot iterator
iterator = dataset.make_one_shot_iterator()
data = iterator.get_next()   
....
for i in range(epochs):
    for ii in range(iteration):
        image = sess.run(data)
        ....
     saver.save(sess, 'filename')

错误消息如下所示

[libprotobuf FATAL external/protobuf_archive/src/google/protobuf/message_lite.cc:68] CHECK failed: (byte_size_before_serialization) == (byte_size_after_serialization): tensorflow.GraphDef was modified concurrently during serialization.
terminate called after throwing an instance of 'google::protobuf::FatalException'  
what():  CHECK failed: (byte_size_before_serialization) == (byte_size_after_serialization): tensorflow.GraphDef was modified concurrently during serialization.
Aborted

Answer 1

问题似乎在 _parse_function。确保解析器在您创建 TFrecord 文件时以相同的方式进行操作。例如，如果它们具有相同的数据类型或

由于保存模型导致训练崩溃："tensorflow.GraphDef was modified concurrently during serialization"

Training crash due to saving model: "tensorflow.GraphDef was modified concurrently during serialization"

python

machine-learning

deep-learning

tensorflow

tensorflow-datasets