Accuracy paradox

The accuracy paradox is the paradoxical finding that accuracy is not a good metric for predictive models when classifying in predictive analytics. This is because a simple model may have a high level of accuracy but too crude to be useful. For example, if the incidence of category A is dominant, being found in 99% of cases, then predicting that every case is category A will have an accuracy of 99%. Precision and recall are better measures in such cases. The underlying issue is that there is a class imbalance between the positive class and the negative class. Prior probabilities for these classes need to be accounted for in error analysis. Precision and recall help, but precision too can be biased by unbalanced class priors in the test sets.[citation needed]

Example

A profiling system to report potential terrorists should not produce too many false positives or the people reviewing the results will have too much useless work to do. For example, a city of 1 million people has ten terrorists. Call civilians the non-terrorists for brevity. A profiling system reports the 10 terrorists, plus 990 false positives, and no false negatives, giving the following confusion matrix:

Predicted
class
Actual class
TCSum
T 10010
C 9901 million - 10001 million - 10
Sum 10001 million - 10001 million

Even though the accuracy is ⁠10 + (1 million - 1000)/1 million⁠ ≈ 99.9%, 990 out of the 1000 positive predictions are incorrect. The precision of ⁠10/10 + 990⁠ = 1% reveals its poor performance. As the classes are so unbalanced, a better metric is the F1 score = ⁠2 × 0.01 × 1/0.01 + 1⁠ ≈ 2% (the recall being ⁠10 + 0/10⁠ = 1).

Literature

See also