Micro-accuracy is generally better aligned with the business needs of ML predictions. If we are only choosing one metric to report for a multiclass classification task, it should be micro-accuracy.
Example, for a support ticket classification task: (maps incoming tickets to teams)
- Micro-accuracy -- how often does an incoming ticket get classified to the right team?
- Macro-accuracy -- for an average team, how often is an incoming ticket correct for their team?
Macro-accuracy overweights small teams in this example; a small team which gets only 10 tickets per year counts as much as a large team with 10k tickets per year. Micro-accuracy in this case correlates better with the business need of, "how much time/money can the company save by automating my ticket routing process".
Below we are reporting only macro-accuracy:
|
nameof(ClassificationMetrics.AccuracyMacro), |
|
_metrics.AccuracyMacro.ToString("0.##", CultureInfo.InvariantCulture)); |
Benchmark output: (src)
Method | Mean | Error | StdDev | Extra Metric |
-------------------- |-------------:|-----------:|-----------:|--------------------:|
PredictIris | 1.650 ms | 0.0151 ms | 0.0141 ms | AccuracyMacro: 0.98 |
PredictIrisBatchOf1 | 1.599 ms | 0.0362 ms | 0.0339 ms | AccuracyMacro: 0.98 |
PredictIrisBatchOf2 | 1.646 ms | 0.0179 ms | 0.0167 ms | AccuracyMacro: 0.98 |
PredictIrisBatchOf5 | 1.635 ms | 0.0192 ms | 0.0179 ms | AccuracyMacro: 0.98 |
For this Iris dataset benchmark, the difference between macro/micro-accuracy are non-important, though it sets a bad precedent which will be replicated in further benchmarks.
Work:
- The above test should be changed to report micro-accuracy
- See if other benchmarks are reporting macro-accuracy
- (future) Report additional metrics instead of just one
Micro-accuracy is generally better aligned with the business needs of ML predictions. If we are only choosing one metric to report for a multiclass classification task, it should be micro-accuracy.
Example, for a support ticket classification task: (maps incoming tickets to teams)
Macro-accuracy overweights small teams in this example; a small team which gets only 10 tickets per year counts as much as a large team with 10k tickets per year. Micro-accuracy in this case correlates better with the business need of, "how much time/money can the company save by automating my ticket routing process".
Below we are reporting only macro-accuracy:
machinelearning/test/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs
Lines 42 to 43 in 23659b0
Benchmark output: (src)
For this Iris dataset benchmark, the difference between macro/micro-accuracy are non-important, though it sets a bad precedent which will be replicated in further benchmarks.
Work: