ML.Net 准确率始终为 100%

Question

我正在使用来自 Kaggle 的 Question Pairs Dataset 和 SdcaLogisticRegression。 ML.Net的版本是14.0

我的电脑规格：

1. 操作系统 Microsoft Windows 10 Pro
2. Systemtyp x64-basierter PC
3. RAM 32,0 GB
4. CPU Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz, 2208 MHz, 6 Kern(e), 12 logische(r) Prozessor(en)

Program.cs:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.DataOperationsCatalog;
using Microsoft.ML.Trainers;
using Microsoft.ML.Transforms.Text;

namespace Csharp_machieneLearning
{
    class Program
    {
        private static IDataView TransData;

        public static void Evaluate(MLContext mlContext, ITransformer model, IDataView splitTestSet)
        {
            Console.WriteLine("=============== Evaluating Model accuracy with Test data===============");
            IDataView predictions = model.Transform(splitTestSet);
            CalibratedBinaryClassificationMetrics metrics = mlContext.BinaryClassification.Evaluate(predictions, "is_duplicate");
            Console.WriteLine();
            Console.WriteLine("Model quality metrics evaluation");
            Console.WriteLine("--------------------------------");
            Console.WriteLine($"Accuracy: {metrics.Accuracy:P2}");
            Console.WriteLine($"Auc: {metrics.AreaUnderRocCurve:P2}");
            Console.WriteLine($"F1Score: {metrics.F1Score:P2}");
            Console.WriteLine("=============== End of model evaluation ===============");
        }


        static void Main(string[] args)
        {
            MLContext mlContext = new MLContext();
            Console.WriteLine($"=============== Loading Dataset  ===============");
            IDataView file = mlContext.Data.LoadFromTextFile<QuestionPairs>(@"C:\Users\ludwi\source\repos\Csharp_machieneLearning\questions.csv", separatorChar: ',', hasHeader: true);
            Console.WriteLine($"=============== Finished Loading Dataset  ===============");
            IEstimator<ITransformer> pipeline = mlContext.Transforms.Conversion.ConvertType("is_duplicate", outputKind: DataKind.Boolean)
                            //.Append(mlContext.Transforms.Conversion.MapValueToKey(inputColumnName: "is_duplicate", outputColumnName: "Label"))
                            .Append(mlContext.Transforms.Text.FeaturizeText(inputColumnName: "question1", outputColumnName: "question1Featurized"))
                            .Append(mlContext.Transforms.Text.FeaturizeText(inputColumnName: "question2", outputColumnName: "question2Featurized"))
                            .Append(mlContext.Transforms.Concatenate("Features", "question1Featurized", "question2Featurized"))
                            .Append(mlContext.Transforms.NormalizeMinMax("Features"));

            IEstimator<ITransformer> estimator = mlContext.BinaryClassification.Trainers.SdcaLogisticRegression(labelColumnName: "is_duplicate", featureColumnName: "Features");

            var transData = pipeline.Fit(file).Transform(file);
            var data = mlContext.Data.TrainTestSplit(transData, testFraction: 0.25);
            var model = estimator.Fit(data.TrainSet);

            Evaluate(mlContext, model, data.TestSet);
        }
    }
}

QuestionPairs.cs:

using System;
using System.Collections.Generic;
using System.Dynamic;
using System.Text;
using Microsoft.ML.Data;

namespace Csharp_machieneLearning
{
    public class QuestionPairs
    {
        [LoadColumn(3)]
        public string question1 { get; set; }

        [LoadColumn(4)]
        public string question2 { get; set; }

        [LoadColumn(5)]
        public string is_duplicate { get; set; }

    }
    public class QuestionPrediction : QuestionPairs
    {
        [ColumnName("PredictedLabel")]
        public bool Prediction { get; set; }

        public float Probability { get; set; }

        public float Score { get; set; }
    }
}

输出：

Answer 1

I though, the problem could be the ConvertType("is_duplicate", outputKind: DataKind.Boolean), so I created a custom Transformer:

Action<QuestionPairs, transformOutput> mapping = (input, output) => { output.Label = input.is_duplicate.Equals("1") ? true : false; };
            IEstimator<ITransformer> pipeline = mlContext.Transforms.CustomMapping(mapping, contractName: null)
                            .Append(mlContext.Transforms.Text.FeaturizeText(inputColumnName: "question1", outputColumnName: "question1Featurized"))
                            .Append(mlContext.Transforms.Text.FeaturizeText(inputColumnName: "question2", outputColumnName: "question2Featurized"))
                            .Append(mlContext.Transforms.Concatenate("Features", "question1Featurized", "question2Featurized"))
                            //.Append(mlContext.Transforms.NormalizeMinMax("Features"))
                            //.AppendCacheCheckpoint(mlContext)
                            .Append(mlContext.BinaryClassification.Trainers.SdcaLogisticRegression(labelColumnName: nameof(customTransform.Label), featureColumnName: "Features"));

这似乎没有帮助，所以我想知道程序是否完全正确地加载了数据集。

Therfor I added a Preview()function.

var file = pipeline.Preview(10);

            foreach(var row in preview.RowView)
            {
                foreach(var column in row.Values)
                {
                    Console.WriteLine(column);
                }
                Console.WriteLine("=============================================================");
            }

Output: As you can see, the "is_dublicat" column sometimes contains a string which is meant to be part of the features. This is caused by an "," used in the feature sentence.

快速搜索后，我找到了 LoadFromTextFile() 函数的 allowQuoting: true 属性，结果似乎是这样的： 运行完整代码后，结果如其所愿。