为什么我在 tensorboard t-sne 中可视化的 cnn 图像特征是随机的？

Question

我有一个卷积神经网络 (VGG16)，它在 26 张图像的分类任务上表现良好类。现在我想在张量板上使用 t-SNE 可视化数据分布。我删除了 CNN 的最后一层，因此输出是 4096 个特征。因为分类工作正常（~90% val_accuracy），我希望在 t-SNE 中看到类似模式的东西。但无论我做什么，分布都保持随机（-> 数据在 circle/sphere 中对齐，类是杂乱无章的）。我做错什么了吗？我是否误解了 t-SNE 或 tensorboard？这是我第一次使用它。

这是我获取功能的代码：

import tensorflow as tf
from tensorflow.keras import Model
import os
import numpy as np
from tensorboard.plugins import projector


def get_image_features(data_dir, model, layername='fc2'):  # fc2 = last vgg dense layer

    out = model.get_layer(layername).output
    feature_model = Model(model.input, out)   # the model, see summary() below

    dataGen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1. / model.input.shape[2])
    test_batches = dataGen.flow_from_directory(data_dir, target_size=model.input.shape[1:3], class_mode='categorical',
                                               batch_size=50, shuffle=False)
    features = feature_model.predict(test_batches)

    return features


def create_projector(self, features, metadata_path):
    import tensorflow.compat.v1 as tf   # tf v1 needed
    tf.disable_v2_behavior()

    if type(features) is str:
        features = np.loadtxt(features)

    data_dir_list = os.listdir(self.test_data_dir)
    data_dir_list.sort()
    img_data = []
    for dataset in data_dir_list:
        img_list = os.listdir(os.path.join(self.test_data_dir, dataset))
        img_list.sort()
        for img in img_list:
            input_img = cv2.imread(os.path.join(self.test_data_dir, dataset, img))
            input_img_resize = cv2.resize(input_img, (112, 112))
            img_data.append(input_img_resize)

    img_data = np.array(img_data)
    features_var = tf.Variable(features, name='features')

    with tf.Session() as sess:
        saver = tf.train.Saver([features_var])

        sess.run(features_var.initializer)
        saver.save(sess, os.path.join(self.log_dir, 'images_4_classes.ckpt'))

        config = projector.ProjectorConfig()
        embedding = config.embeddings.add()
        embedding.tensor_name = features_var.name
        embedding.metadata_path = metadata_path

        projector.visualize_embeddings(tf.summary.FileWriter(self.log_dir), config)

编辑：当我取消选中“spherize data”时，它不再是一个球体，但类的分布似乎仍然是完全随机的。顺便说一句：PCA 工作正常并显示某种模式

这是 feature_model.summary():

Layer (type)                 Output Shape              Param #
=================================================================
block1_conv1_input (InputLay [(None, 224, 224, 3)]     0
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 56, 56, 128)       0
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 56, 56, 256)       295168
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 56, 56, 256)       590080
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 56, 56, 256)       590080
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 28, 28, 256)       0
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 28, 28, 512)       1180160
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 28, 28, 512)       2359808
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 28, 28, 512)       2359808
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, 14, 14, 512)       0
_________________________________________________________________
block5_conv1 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_conv2 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_conv3 (Conv2D)        (None, 14, 14, 512)       2359808
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, 7, 7, 512)         0
_________________________________________________________________
flatten (Flatten)            (None, 25088)             0
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              102764544
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312
=================================================================

Answer 1

几周后，我不再尝试使用 tensorboard。我将输出层中的特征数量减少到 256、128、64，我之前使用 PCA 和 Truncated SDV 减少了特征，但没有任何改变。

现在我使用 sklearn.manifold.TSNE 并使用 plotly 可视化输出。这也很简单，工作正常，我可以看到适当的模式，而张量板中的 t-SNE 仍然产生随机分布。所以我猜 tensorboard 中的算法太多了类。或者我在准备数据时犯了一个错误并且没有注意到（但是为什么 PCA 起作用？）

如果有人知道问题出在哪里，我仍然很好奇。但如果其他人遇到同样的问题，我建议您使用 sklearn 进行尝试。

为什么我在 tensorboard t-sne 中可视化的 cnn 图像特征是随机的？

why is my visualization of cnn image features in tensorboard t-sne RANDOM?

python

tensorflow

keras

conv-neural-network

tensorboard