Classification ModelOutput.Score values' range are inconsistent

[andrasfuchs]

System Information (please complete the following information):

• Model Builder Version (available in Manage Extensions dialog): 16.13.1.2210302
• Visual Studio Version: 17.0.5

Describe the bug

• On which step of the process did you run into an issue:
  ML model consumption

• Clear description of the problem:
  I tested various ML models with my input that had a few features and one classification label.

The ModelOutput was generated successfully, but the Score values had the following ranges depending the the algorithm:
FastForestBinary [-4.00..+4.00]
LightGbmBinary [-9.72..+9.01]
SdcaLogisticRegressionBinary [+0.03..+0.63]

I would have expected to have the same normalized range for all of them, so the Score value would have been between 0 and 1 depending on the confidence level of models.

To Reproduce
Steps to reproduce the behavior:

1. Create a new .NET 6 project
2. Add a new machine learning model and start Model Builder
3. Set a CSV dataset as input with a few feature columns and one classification label column
4. Start the training, and stop it at different algorithms so that you can test the behavior
5. Run the generated code and feed different inputs to the ModelInput and run the Predict method
6. Check the range of Score property on the ModelOutput class

Expected behavior
I would expect to have Score values between 0.0 and 1.0 indicating the confidence of the model.

[JakeRadMSFT]

@michaelgsharp @LittleLittleCloud do we have a way to normalize the scores?

[LittleLittleCloud]

The inconsistency of range for scores from different trainers is expected. Since score is essentially the output of a trainer's optimize function, its range is mainly determined by that function. Some trainers has a linear function, which often results to a evenly distributed result in score, while some other trainers' are not linear, in which case the distributions of those scores are also often various. Usually a comparison between scores from different trainers doesn't have any meaningful value, which means, a higher score doesn't mean a better result/confidence

In this case, since both LGBM and FastForest are tree-based method, it's expected to see that their score has +/- value and symmetric to 0. While Sdca is a linear trainer, its score should also be close to a linear distribution.

You can find out the normalized value in Probability which is always between (0.1), but there's little meaning to compare those values from different trainers as well.

[andrasfuchs]

@LittleLittleCloud Thank you for the explanation, I didn't know how the Score is calculated!

What is the practical use then of this Score value in the case of a classification scenario?

Could you also explain why the Probability property gets only generated if the IsCategorical value is false on the label in .mbconfig?

[LittleLittleCloud]

(For binary classification case only)
Score will be used as input in calibrator, which calculates Probability for all labels. You can check how ICalibrator is defined here

I don't quite understand the second question, do you mean that the Probability doesn't appear if IsCategorical value is false?

[andrasfuchs]

Thank you!

There is no Probability property on the ModelOutput class if the IsCategorical value is set to true on the label.
If it set to false then the property is there. I use Model Builder 2022 v16.13.1.2210302.

[LittleLittleCloud]

Close this thread and tracking it in dotnet/machinelearning#6120