使用 ML5 训练的模型中的 tfjs 进行预测时准确度降低

Question

我在 Google Chrome 上使用 tfjs 1.0.0 | 76.0.3809.132（正式版）（64 位）

我在项目中使用 ML5 训练图像分类模型。我使用特征提取器进行迁移学习。我使用 mobilenet_v1_0.25 作为基础模型。我想集成它，以便它从 chrome 扩展执行预测。我不得不使用 tfjs，因为我发现 ML5 没有从扩展程序的后台页面运行。我用tfjs加载了ML5训练好的模型，然后开始预测。但是，与在 ML5 本身中使用相同模型进行预测时相比，tfjs 中的预测准确度非常低。

我尝试通过废弃 ML5 Feature Extractor 源代码在 tfjs 中重现 ML5 的预测，但是，从 tfjs 进行预测时，预测精度仍然大大降低。

我先加载mobilenet和自定义模型做一个联合模型

load() {
    console.log("ML Data loading..");
    // ! ==========================================
    // ! This is a work around and will only work for default version and alpha values that were used while training model.
    this.mobilenet = await tf.loadLayersModel("https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_0.25_224/model.json");
    const layer = this.mobilenet.getLayer('conv_pw_13_relu');
    // ! ==========================================

    this.mobilenetFeatures = await tf.model({ inputs: this.mobilenet.inputs, outputs: layer.output });
    this.customModel = await tf.loadLayersModel("./model.json");
    this.model.add(this.mobilenetFeatures);
    this.model.add(this.customModel);
}

然后我将图像传递给一个函数，该函数在预测

后获得最高值类

let result = this.getTopKClasses(this.predict(image), 5)

getTopKClasses(logits, topK) {
    const predictions = logits;
    const values = predictions.dataSync();
    predictions.dispose();
    let predictionList = [];
    for (let i = 0; i < values.length; i++) {
        predictionList.push({ value: values[i], index: i });
    }
    predictionList = predictionList
        .sort((a, b) => {
            return b.value - a.value;
        })
        .slice(0, topK);
    console.log(predictionList);
    let site = predictionList[0];
    let result = { type: 'custom', site: IMAGENET_CLASSES[site.index] }
    console.log('ML Result: Site: %s, Probability: %i%', result.site, (site.value * 100));
    if (site.value > ML_THRESHOLD) {
        return result;
    } else {
        return null;
    }
}

predict(image) {
    const preprocessed = this.imgToTensor(image, [224, 224])
    console.log(preprocessed);
    var result = this.model.predict(preprocessed);
    return result;
}

辅助函数：

imgToTensor(input, size = null) {
    return tf.tidy(() => {
        let img = tf.browser.fromPixels(input);
        if (size) {
            img = tf.image.resizeBilinear(img, size);
        }
        const croppedImage = this.cropImage(img);
        const batchedImage = croppedImage.expandDims(0);
        return batchedImage.toFloat().div(tf.scalar(127)).sub(tf.scalar(1));
    });
}

cropImage(img) {
    const size = Math.min(img.shape[0], img.shape[1]);
    const centerHeight = img.shape[0] / 2;
    const beginHeight = centerHeight - (size / 2);
    const centerWidth = img.shape[1] / 2;
    const beginWidth = centerWidth - (size / 2);
    return img.slice([beginHeight, beginWidth, 0], [size, size, 3]);
};

Answer 1

为了从使用迁移学习训练的模型（即在另一个预训练模型上训练）进行预测，您需要首先从基础模型进行预测，然后通过传递基础模型从自定义模型进行预测-模型预测张量到自定义模型预测的输入。

async load(options = {}) {

    // Loading the layer from the base model.
    this.mobilenet = await tf.loadLayersModel(`${BASE_URL}${this.config.version}_${this.config.alpha}_${IMAGE_SIZE}/model.json`);
    const layer = this.mobilenet.getLayer(this.config.layer);

    //Converting the base-model layer to a model.
    this.mobilenetFeatures = await tf.model({ inputs: this.mobilenet.inputs, outputs: layer.output });

    // Loading the custom model that was trained by us.
    this.customModel = await tf.loadLayersModel(CUSTOM_MODEL_FILE_URL);
}

现在根据这些模型进行预测：

predict(image) {
    // Converting image to tensor
    const preprocessed = this.imgToTensor(image, [224, 224])

    // * Make predictions about the image firstly, from the Mobilenet (base) Model.
    const embeddings = this.mobilenetFeatures.predict(preprocessed);

    // * Filter predictions from Mobilenet Model using custom trained Model.
    const result = this.customModel.predict(embeddings);

    return result;
}

使用 ML5 训练的模型中的 tfjs 进行预测时准确度降低

Accuracy reduced when predicting using tfjs from model trained by ML5

javascript

machine-learning

tensorflow

tensor