尝试连接两个模型并适合 Keras 时出现 AssertionError

Question

我正在尝试开发图像字幕模型。我指的是这个 Github repository。我有三种方法，它们执行以下操作：

生成图像模型
生成字幕模型
将图像和标题模型连接在一起

由于代码较长，我创建了一个Gist to show the methods。

这里是summary of my image model and caption model.

但是后来我运行代码，我收到了这个错误：

TraceTraceback (most recent call last):
  File "trainer.py", line 99, in <module>
    model.fit([images, encoded_captions], one_hot_captions, batch_size = 1, epochs = 5)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/engine/training.py", line 950, in fit
    batch_size=batch_size)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/engine/training.py", line 671, in _standardize_user_data
    self._set_inputs(x)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/keras/engine/training.py", line 575, in _set_inputs
    assert len(inputs) == 1
AssertionError

由于错误来自 Keras 库，我不知道如何调试它。但是当我尝试将它们连接在一起时出了点问题。

我想知道我是否遗漏了什么

Answer 1

您需要使用 output 属性获取模型的输出，然后使用 Keras functional API to be able to concatenate them (by either of Concatenate layer or its equivalent functional interface concatenate) 并创建最终模型：

from keras.models import Model

image_model = get_image_model()
language_model = get_language_model(vocab_size)

merged = concatenate([image_model.output, language_model.output])
x = LSTM(256, return_sequences = False)(merged)
x = Dense(vocab_size)(x)
out = Activation('softmax')(x)

model = Model([image_model.input, language_model.input], out)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
model.fit([images, encoded_captions], one_hot_captions, ...)

就像现在在您的代码中一样，您还可以为模型创建逻辑定义一个函数：

def get_concatenated_model(image_model, language_model, vocab_size):
    merged = concatenate([image_model.output, language_model.output])
    x = LSTM(256, return_sequences = False)(merged)
    x = Dense(vocab_size)(x)
    out = Activation('softmax')(x)

    model = Model([image_model.input, language_model.input], out)
    return model

尝试连接两个模型并适合 Keras 时出现 AssertionError

AssertionError while trying to concatenate two models and fit in Keras

python

machine-learning

deep-learning

keras

keras-layer