如何训练和测试从kaggle下载的图像数据集
how to train amd test dataset of images downloaded from kaggle
我想从 Kaggle 加载数据集。数据集的 link 是 https://www.kaggle.com/sagyamthapa/handwritten-math-symbols.
它在不同的文件夹中有图像。如何标记数据集并拆分和训练它。
我是按照下面的方法做的，但是报错了
train_ds = tf.keras.preprocessing.image_dataset_from_directory(data_dir,#color_mode="grayscale",validation_split=0.2,subset="training",seed=123,image_size=(img_height, img_width),batch_size=batch_size)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(data_dir,#color_mode="grayscale",validation_split=0.2,subset="validation",seed=123,image_size=(img_height, img_width),batch_size=batch_size)
代码如下
import os
import pandas as pd
from sklearn.model_selection import train_test_split
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Dense, Activation,Dropout,Conv2D, MaxPooling2D,BatchNormalization
from tensorflow.keras.optimizers import Adam, Adamax
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras import regularizers
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Model, load_model, Sequential
sdir=r'../input/handwritten-math-symbols/dataset'
classlist=os.listdir(sdir)
filepaths=[]
labels=[]
classes=[]
for klass in classlist:
classpath=os.path.join(sdir, klass)
if os.path.isdir(classpath):
classes.append(klass)
flist=os.listdir(classpath)
for f in flist:
fpath=os.path.join(classpath,f)
if os.path.isfile(fpath):
filepaths.append(fpath)
labels.append(klass)
fseries=pd.Series(filepaths, name='filepaths')
Lseries=pd.Series (labels, name='labels')
df=pd.concat([fseries, Lseries], axis=1)
balance=df['labels'].value_counts()
print (balance) # dataset is reasonably balanced
train_split=.9
test_split=.05
dummy_split=test_split/(1-train_split)
train_df, dummy_df=train_test_split(df, train_size=train_split, shuffle=True, random_state = 123)
test_df, valid_df=train_test_split(dummy_df, train_size=dummy_split, shuffle=True, random_state=123)
def scalar(img):
return img/127.5-1 # scale pixels between -1 and + 1
gen=ImageDataGenerator(preprocessing_function=scalar)
train_gen=gen.flow_from_dataframe(train_df, x_col= 'filepaths', y_col='labels', target_size=(128,128), class_mode='categorical',
color_mode='rgb', shuffle=False)
test_gen=gen.flow_from_dataframe(test_df, x_col= 'filepaths', y_col='labels', target_size=(128,128), class_mode='categorical',
color_mode='rgb', shuffle=False)
valid_gen=gen.flow_from_dataframe(valid_df, x_col= 'filepaths', y_col='labels', target_size=(128,128), class_mode='categorical',
color_mode='rgb', shuffle=False)
base_model=tf.keras.applications.MobileNetV2( include_top=False, input_shape=(128,128,3), pooling='max', weights='imagenet')
x=base_model.output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
x = Dense(1024, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
bias_regularizer=regularizers.l1(0.006) ,activation='relu', kernel_initializer= tf.keras.initializers.GlorotUniform(seed=123))(x)
x=Dropout(rate=.3, seed=123)(x)
output=Dense(len(classes), activation='softmax',kernel_initializer=tf.keras.initializers.GlorotUniform(seed=123))(x)
model=Model(inputs=base_model.input, outputs=output)
model.compile(Adamax(lr=.001), loss='categorical_crossentropy', metrics=['accuracy'])
estop=tf.keras.callbacks.EarlyStopping( monitor="val_loss", patience=4, verbose=1,restore_best_weights=True)
rlronp=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss",factor=0.5, patience=1, verbose=1)
history=model.fit(x=train_gen, epochs=10, verbose=1, callbacks=[estop, rlronp], validation_data=valid_gen,
validation_steps=None, shuffle=False, initial_epoch=0)
save_path=r'c:\mydir\mymodel.h5' # specify the path to where to save model
model.save(save_path)
model.fit的结果应该如下图
Epoch 1/20
254/254 [==============================] - 26s 84ms/step - loss: 14.9756 - accuracy: 0.8516 - val_loss: 5.0730 - val_accuracy: 0.6452
Epoch 2/20
254/254 [==============================] - 18s 73ms/step - loss: 2.7752 - accuracy: 0.9945 - val_loss: 1.7161 - val_accuracy: 0.7783
Epoch 3/20
254/254 [==============================] - 20s 78ms/step - loss: 0.7500 - accuracy: 0.9994 - val_loss: 0.9572 - val_accuracy: 0.8780
Epoch 4/20
254/254 [==============================] - 21s 84ms/step - loss: 0.3855 - accuracy: 0.9998 - val_loss: 0.6381 - val_accuracy: 0.9357
Epoch 5/20
254/254 [==============================] - 18s 71ms/step - loss: 0.2984 - accuracy: 1.0000 - val_loss: 0.4525 - val_accuracy: 0.9601
Epoch 6/20
254/254 [==============================] - 18s 73ms/step - loss: 0.2609 - accuracy: 1.0000 - val_loss: 0.3453 - val_accuracy: 0.9778
Epoch 7/20
254/254 [==============================] - 18s 70ms/step - loss: 0.2354 - accuracy: 0.9998 - val_loss: 0.2760 - val_accuracy: 0.9867
Epoch 8/20
254/254 [==============================] - 18s 69ms/step - loss: 0.2160 - accuracy: 1.0000 - val_loss: 0.2478 - val_accuracy: 0.9911
Epoch 9/20
254/254 [==============================] - 18s 70ms/step - loss: 0.2023 - accuracy: 1.0000 - val_loss: 0.2042 - val_accuracy: 0.9956
Epoch 10/20
254/254 [==============================] - 19s 74ms/step - loss: 0.1894 - accuracy: 1.0000 - val_loss: 0.1889 - val_accuracy: 0.9956
我想从 Kaggle 加载数据集。数据集的 link 是 https://www.kaggle.com/sagyamthapa/handwritten-math-symbols.
它在不同的文件夹中有图像。如何标记数据集并拆分和训练它。
我是按照下面的方法做的，但是报错了
train_ds = tf.keras.preprocessing.image_dataset_from_directory(data_dir,#color_mode="grayscale",validation_split=0.2,subset="training",seed=123,image_size=(img_height, img_width),batch_size=batch_size)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(data_dir,#color_mode="grayscale",validation_split=0.2,subset="validation",seed=123,image_size=(img_height, img_width),batch_size=batch_size)
代码如下
import os
import pandas as pd
from sklearn.model_selection import train_test_split
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import Dense, Activation,Dropout,Conv2D, MaxPooling2D,BatchNormalization
from tensorflow.keras.optimizers import Adam, Adamax
from tensorflow.keras.metrics import categorical_crossentropy
from tensorflow.keras import regularizers
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Model, load_model, Sequential
sdir=r'../input/handwritten-math-symbols/dataset'
classlist=os.listdir(sdir)
filepaths=[]
labels=[]
classes=[]
for klass in classlist:
classpath=os.path.join(sdir, klass)
if os.path.isdir(classpath):
classes.append(klass)
flist=os.listdir(classpath)
for f in flist:
fpath=os.path.join(classpath,f)
if os.path.isfile(fpath):
filepaths.append(fpath)
labels.append(klass)
fseries=pd.Series(filepaths, name='filepaths')
Lseries=pd.Series (labels, name='labels')
df=pd.concat([fseries, Lseries], axis=1)
balance=df['labels'].value_counts()
print (balance) # dataset is reasonably balanced
train_split=.9
test_split=.05
dummy_split=test_split/(1-train_split)
train_df, dummy_df=train_test_split(df, train_size=train_split, shuffle=True, random_state = 123)
test_df, valid_df=train_test_split(dummy_df, train_size=dummy_split, shuffle=True, random_state=123)
def scalar(img):
return img/127.5-1 # scale pixels between -1 and + 1
gen=ImageDataGenerator(preprocessing_function=scalar)
train_gen=gen.flow_from_dataframe(train_df, x_col= 'filepaths', y_col='labels', target_size=(128,128), class_mode='categorical',
color_mode='rgb', shuffle=False)
test_gen=gen.flow_from_dataframe(test_df, x_col= 'filepaths', y_col='labels', target_size=(128,128), class_mode='categorical',
color_mode='rgb', shuffle=False)
valid_gen=gen.flow_from_dataframe(valid_df, x_col= 'filepaths', y_col='labels', target_size=(128,128), class_mode='categorical',
color_mode='rgb', shuffle=False)
base_model=tf.keras.applications.MobileNetV2( include_top=False, input_shape=(128,128,3), pooling='max', weights='imagenet')
x=base_model.output
x=keras.layers.BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001 )(x)
x = Dense(1024, kernel_regularizer = regularizers.l2(l = 0.016),activity_regularizer=regularizers.l1(0.006),
bias_regularizer=regularizers.l1(0.006) ,activation='relu', kernel_initializer= tf.keras.initializers.GlorotUniform(seed=123))(x)
x=Dropout(rate=.3, seed=123)(x)
output=Dense(len(classes), activation='softmax',kernel_initializer=tf.keras.initializers.GlorotUniform(seed=123))(x)
model=Model(inputs=base_model.input, outputs=output)
model.compile(Adamax(lr=.001), loss='categorical_crossentropy', metrics=['accuracy'])
estop=tf.keras.callbacks.EarlyStopping( monitor="val_loss", patience=4, verbose=1,restore_best_weights=True)
rlronp=tf.keras.callbacks.ReduceLROnPlateau( monitor="val_loss",factor=0.5, patience=1, verbose=1)
history=model.fit(x=train_gen, epochs=10, verbose=1, callbacks=[estop, rlronp], validation_data=valid_gen,
validation_steps=None, shuffle=False, initial_epoch=0)
save_path=r'c:\mydir\mymodel.h5' # specify the path to where to save model
model.save(save_path)
model.fit的结果应该如下图
Epoch 1/20
254/254 [==============================] - 26s 84ms/step - loss: 14.9756 - accuracy: 0.8516 - val_loss: 5.0730 - val_accuracy: 0.6452
Epoch 2/20
254/254 [==============================] - 18s 73ms/step - loss: 2.7752 - accuracy: 0.9945 - val_loss: 1.7161 - val_accuracy: 0.7783
Epoch 3/20
254/254 [==============================] - 20s 78ms/step - loss: 0.7500 - accuracy: 0.9994 - val_loss: 0.9572 - val_accuracy: 0.8780
Epoch 4/20
254/254 [==============================] - 21s 84ms/step - loss: 0.3855 - accuracy: 0.9998 - val_loss: 0.6381 - val_accuracy: 0.9357
Epoch 5/20
254/254 [==============================] - 18s 71ms/step - loss: 0.2984 - accuracy: 1.0000 - val_loss: 0.4525 - val_accuracy: 0.9601
Epoch 6/20
254/254 [==============================] - 18s 73ms/step - loss: 0.2609 - accuracy: 1.0000 - val_loss: 0.3453 - val_accuracy: 0.9778
Epoch 7/20
254/254 [==============================] - 18s 70ms/step - loss: 0.2354 - accuracy: 0.9998 - val_loss: 0.2760 - val_accuracy: 0.9867
Epoch 8/20
254/254 [==============================] - 18s 69ms/step - loss: 0.2160 - accuracy: 1.0000 - val_loss: 0.2478 - val_accuracy: 0.9911
Epoch 9/20
254/254 [==============================] - 18s 70ms/step - loss: 0.2023 - accuracy: 1.0000 - val_loss: 0.2042 - val_accuracy: 0.9956
Epoch 10/20
254/254 [==============================] - 19s 74ms/step - loss: 0.1894 - accuracy: 1.0000 - val_loss: 0.1889 - val_accuracy: 0.9956