Keras 混合功能模型中的图像增强
Image augmentation in Keras mixed functional model
我在 Keras 中创建了一个混合模型，为元数据和图像数据创建权重，然后将它们组合起来进行分类。这是模型：
Model: "model_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_5 (InputLayer) [(None, 80, 120, 3)] 0
__________________________________________________________________________________________________
xception (Functional) (None, 3, 4, 2048) 20861480 input_5[0][0]
__________________________________________________________________________________________________
input_4 (InputLayer) [(None, 10)] 0
__________________________________________________________________________________________________
conv2d_9 (Conv2D) (None, 3, 4, 8) 409608 xception[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 4) 44 input_4[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 8) 0 conv2d_9[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 12) 0 dense_3[0][0]
global_average_pooling2d_1[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 4) 52 concatenate_1[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 1) 5 dense_4[0][0]
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________
由于不平衡，我决定增强图像。我使用了以下 ImageDataGenerator：
aug = ImageDataGenerator(rescale=1/255.,
rotation_range=180,
height_shift_range=0.2,
width_shift_range=0.2,
brightness_range=[0.5,1.5],
channel_shift_range=100.0,
horizontal_flip=True,
vertical_flip=True,
shear_range=45.0)
然后我编译并尝试使用 ImageDataGenerator().flow():
训练模型
epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train, y_train, batch_size=BATCH_SIZE)
history = model.fit([meta_train, flow], y_train, epochs=epochs, batch_size=100, validation_data=([meta_test, img_test], y_test), class_weight=class_weight)
这给了我一个错误：
ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>", "<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),
<class 'numpy.ndarray'>
我已经尝试了多个版本的代码，但我对后端不够熟悉，无法正确诊断问题。谁能帮我解决这个问题？
模型代码和 MRE
型号代码
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,120,3,))
# Model 1
meta_layer1 = Dense(4, activation='relu')(meta_inputs)
# Model 2
xception_layer = Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)
# Merge models
merged_layer = Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = Dense(4, activation='relu')(merged_layer)
merged_output = Dense(1, activation='sigmoid')(merged_dense_layer)
# Define functional model
model = Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)
# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()
meta_train MRE
age_approx Unknown female male head/neck lower extremity \
11655 45 0 0 1 0 0
24502 60 0 0 1 0 1
2524 50 0 1 0 0 1
13894 60 0 1 0 0 0
29325 45 0 1 0 0 1
oral/genital palms/soles torso upper extremity
11655 0 0 1 0
24502 0 0 0 0
2524 0 0 0 0
13894 0 0 1 0
29325 0 0 0 0
img_train MRE
数组太大，请参阅代码 here。
y_train.shape
(23188, 1)
这不是直接的答案，但有助于解决问题。缺少解决您问题的信息。
但首先：最明显的问题是；您提供 meta_train 这是一个 pandas 数据框。将其转换为数组。 请先尝试这个。
第二个，如果你还有问题，那么model.fit可能无法处理一个列表[meta_train, flow]，那么你可能需要想办法提供您的模型要处理的两个输入。
为此，您应该提供以下内容作为 MRE 以重现您的问题。
(1) 您的模型代码，即使根据提供的摘要可能会生成相同的模型。
(2) y_train.
的形状
首先，如果您生成以下数组，您的模型可以正常工作；
import tensorflow as tf
import numpy as np
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = tf.keras.layers.Input(shape=(10,))
img_inputs = tf.keras.layers.Input(shape=(80,120,3,))
# Model 1
meta_layer1 = tf.keras.layers.Dense(4, activation='relu')(meta_inputs)
# Model 2
xception_layer = tf.keras.applications.Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = tf.keras.layers.Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = tf.keras.layers.GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)
# Merge models
merged_layer = tf.keras.layers.Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = tf.keras.layers.Dense(4, activation='relu')(merged_layer)
merged_output = tf.keras.layers.Dense(1, activation='sigmoid')(merged_dense_layer)
# Define functional model
model = tf.keras.models.Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)
# Compile model
auc = tf.keras.metrics.AUC(name = 'auc')
model.compile(tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()
让我们生成如下数组；
y_array = np.zeros((23188, 1))
train_array = np.zeros((23188, 80, 120,3))
meta_array = np.zeros((23188, 10))
现在测试你的模型；
model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
如您所见，它在运行
>>> model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
2021-05-17 10:30:03.430303: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-05-17 10:30:03.447843: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3093120000 Hz
47/725 [>.............................] - ETA: 6:52 - loss: 0.6818 - auc: 0.0000e+00
现在问题出在您的输入中：
(1) 首先将所有元数据转换为一个数组，假设它有 1000 个批次，因此它必须具有 meta_train.shape 的形状是 (1000,10)
(2) 你的 img_train.shape 是 (1000,80,120,3)
(3) 你的 y_train.shape 是 (1000,1)
这里1000也可以是23188。但我们假设您有 1000 张图片，y_train（目标）和 1000 个元数据。
因为你想增加你的图像序列，所以你必须小心。如下使用；
import tensorflow as tf
import numpy as np
import time
这里我创建了一个空数组作为示例，但是你的原始数据必须按照下面的形状放在数组中，除了不是 1000，它必须是你拥有的图像数量。
y_train = np.zeros((1000, 1))
img_train = np.zeros((1000, 80, 120,3))
meta_train= np.zeros((1000, 10))
aug = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,
rotation_range=180,
height_shift_range=0.2,
width_shift_range=0.2,
brightness_range=[0.5,1.5],
channel_shift_range=100.0,
horizontal_flip=True,
vertical_flip=True,
shear_range=45.0)
aug.fit(img_train)
# NOTE; change batch_size to 32 if your img_train is too high or your device has less memory
# set shuffle to false, you have to concatenate at each loop the meta_train and y_train for each image, so do not shuffle the images!
imagenerator = aug.flow(img_train, batch_size = img_train.shape[0], shuffle=False, sample_weight=None, seed=123, save_to_dir=None, save_prefix="", save_format="png", subset=None)
new_meta = meta_train.copy() # concatenate meta_train at each loop to new_meta
new_y = y_train.copy() # concatenate y_train at each loop to new_y
batches = 0
#let's iterate 5 times.
for x_batch in imagenerator:
print(batches, time.strftime("%Y:%m%d-%H:%M:%S"))
batches += 1
img_train = np.concatenate((img_train, x_batch), axis = 0)
new_meta = np.concatenate((new_meta, meta_train), axis = 0) # concatenate corresponding meta data
new_y = np.concatenate((new_y, y_train), axis = 0) #concatenate corresponding label/target data!
if batches >= 5:
break
model.fit(x = [new_meta, img_train], y = new_y, epochs = 1, batch_size = 32)
注意：在 model.fit
中使用数据之前，您可以按如下方式简单地打乱数据
idxs = np.array([x for x in range(img_train.shape[0])])
np.random.shuffle(idxs)
img_train = img_train[idx]
new_y = new_y[idx]
new_meta = new_meta[idx]
我在 Keras 中创建了一个混合模型，为元数据和图像数据创建权重，然后将它们组合起来进行分类。这是模型：
Model: "model_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_5 (InputLayer) [(None, 80, 120, 3)] 0
__________________________________________________________________________________________________
xception (Functional) (None, 3, 4, 2048) 20861480 input_5[0][0]
__________________________________________________________________________________________________
input_4 (InputLayer) [(None, 10)] 0
__________________________________________________________________________________________________
conv2d_9 (Conv2D) (None, 3, 4, 8) 409608 xception[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 4) 44 input_4[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 8) 0 conv2d_9[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 12) 0 dense_3[0][0]
global_average_pooling2d_1[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None, 4) 52 concatenate_1[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None, 1) 5 dense_4[0][0]
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________
由于不平衡，我决定增强图像。我使用了以下 ImageDataGenerator：
aug = ImageDataGenerator(rescale=1/255.,
rotation_range=180,
height_shift_range=0.2,
width_shift_range=0.2,
brightness_range=[0.5,1.5],
channel_shift_range=100.0,
horizontal_flip=True,
vertical_flip=True,
shear_range=45.0)
然后我编译并尝试使用 ImageDataGenerator().flow():
epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train, y_train, batch_size=BATCH_SIZE)
history = model.fit([meta_train, flow], y_train, epochs=epochs, batch_size=100, validation_data=([meta_test, img_test], y_test), class_weight=class_weight)
这给了我一个错误：
ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>", "<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),
<class 'numpy.ndarray'>
我已经尝试了多个版本的代码，但我对后端不够熟悉，无法正确诊断问题。谁能帮我解决这个问题？
模型代码和 MRE
型号代码
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,120,3,))
# Model 1
meta_layer1 = Dense(4, activation='relu')(meta_inputs)
# Model 2
xception_layer = Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)
# Merge models
merged_layer = Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = Dense(4, activation='relu')(merged_layer)
merged_output = Dense(1, activation='sigmoid')(merged_dense_layer)
# Define functional model
model = Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)
# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()
meta_train MRE
age_approx Unknown female male head/neck lower extremity \
11655 45 0 0 1 0 0
24502 60 0 0 1 0 1
2524 50 0 1 0 0 1
13894 60 0 1 0 0 0
29325 45 0 1 0 0 1
oral/genital palms/soles torso upper extremity
11655 0 0 1 0
24502 0 0 0 0
2524 0 0 0 0
13894 0 0 1 0
29325 0 0 0 0
img_train MRE
数组太大，请参阅代码 here。
y_train.shape
(23188, 1)
这不是直接的答案，但有助于解决问题。缺少解决您问题的信息。
但首先：最明显的问题是；您提供 meta_train 这是一个 pandas 数据框。将其转换为数组。 请先尝试这个。
第二个，如果你还有问题，那么model.fit可能无法处理一个列表[meta_train, flow]，那么你可能需要想办法提供您的模型要处理的两个输入。
为此，您应该提供以下内容作为 MRE 以重现您的问题。
(1) 您的模型代码，即使根据提供的摘要可能会生成相同的模型。
(2) y_train.
的形状首先，如果您生成以下数组，您的模型可以正常工作；
import tensorflow as tf
import numpy as np
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = tf.keras.layers.Input(shape=(10,))
img_inputs = tf.keras.layers.Input(shape=(80,120,3,))
# Model 1
meta_layer1 = tf.keras.layers.Dense(4, activation='relu')(meta_inputs)
# Model 2
xception_layer = tf.keras.applications.Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = tf.keras.layers.Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = tf.keras.layers.GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)
# Merge models
merged_layer = tf.keras.layers.Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = tf.keras.layers.Dense(4, activation='relu')(merged_layer)
merged_output = tf.keras.layers.Dense(1, activation='sigmoid')(merged_dense_layer)
# Define functional model
model = tf.keras.models.Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)
# Compile model
auc = tf.keras.metrics.AUC(name = 'auc')
model.compile(tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()
让我们生成如下数组；
y_array = np.zeros((23188, 1))
train_array = np.zeros((23188, 80, 120,3))
meta_array = np.zeros((23188, 10))
现在测试你的模型；
model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
如您所见，它在运行
>>> model.fit(x = [meta_array, train_array], y = y_array, epochs = 1)
2021-05-17 10:30:03.430303: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-05-17 10:30:03.447843: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3093120000 Hz
47/725 [>.............................] - ETA: 6:52 - loss: 0.6818 - auc: 0.0000e+00
现在问题出在您的输入中：
(1) 首先将所有元数据转换为一个数组，假设它有 1000 个批次，因此它必须具有 meta_train.shape 的形状是 (1000,10)
(2) 你的 img_train.shape 是 (1000,80,120,3)
(3) 你的 y_train.shape 是 (1000,1)
这里1000也可以是23188。但我们假设您有 1000 张图片，y_train（目标）和 1000 个元数据。
因为你想增加你的图像序列，所以你必须小心。如下使用；
import tensorflow as tf
import numpy as np
import time
这里我创建了一个空数组作为示例，但是你的原始数据必须按照下面的形状放在数组中，除了不是 1000，它必须是你拥有的图像数量。
y_train = np.zeros((1000, 1))
img_train = np.zeros((1000, 80, 120,3))
meta_train= np.zeros((1000, 10))
aug = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,
rotation_range=180,
height_shift_range=0.2,
width_shift_range=0.2,
brightness_range=[0.5,1.5],
channel_shift_range=100.0,
horizontal_flip=True,
vertical_flip=True,
shear_range=45.0)
aug.fit(img_train)
# NOTE; change batch_size to 32 if your img_train is too high or your device has less memory
# set shuffle to false, you have to concatenate at each loop the meta_train and y_train for each image, so do not shuffle the images!
imagenerator = aug.flow(img_train, batch_size = img_train.shape[0], shuffle=False, sample_weight=None, seed=123, save_to_dir=None, save_prefix="", save_format="png", subset=None)
new_meta = meta_train.copy() # concatenate meta_train at each loop to new_meta
new_y = y_train.copy() # concatenate y_train at each loop to new_y
batches = 0
#let's iterate 5 times.
for x_batch in imagenerator:
print(batches, time.strftime("%Y:%m%d-%H:%M:%S"))
batches += 1
img_train = np.concatenate((img_train, x_batch), axis = 0)
new_meta = np.concatenate((new_meta, meta_train), axis = 0) # concatenate corresponding meta data
new_y = np.concatenate((new_y, y_train), axis = 0) #concatenate corresponding label/target data!
if batches >= 5:
break
model.fit(x = [new_meta, img_train], y = new_y, epochs = 1, batch_size = 32)
注意：在 model.fit
idxs = np.array([x for x in range(img_train.shape[0])])
np.random.shuffle(idxs)
img_train = img_train[idx]
new_y = new_y[idx]
new_meta = new_meta[idx]