Abstract

The assessment of binary classifier performance traditionally centers on discriminative ability, using metrics such as accuracy. However, these metrics often disregard the model’s inherent uncertainty, an aspect that becomes particularly important in sensitive decision-making domains like healthcare or insurance. For instance, when estimating mortality rates or predicting accident occurrence in an insurance context, since model-predicted scores are often interpreted as event probabilities, having a calibrated model allows for enhancing the reliability of these predictions. While calibration enhances the alignment between predicted scores and actual outcomes, a common misconception is to interpret the resulting scores as true posterior probabilities. Specifically, tree-based classifiers may seem calibrated based on standard calibration metrics and exhibit high predictive performance even if their score distributions do not align with true event distributions. In this study, we analyze how the lack of score heterogeneity in tree-based classifiers can lead to misinterpretation of low calibration errors. Furthermore, since the notion of calibration is often used to assess fairness by comparing errors across groups defined by sensitive attributes such as gender or race, a lack of score heterogeneity within minority groups can conceal underlying disparities. Addressing this issue allows for a more accurate evaluation of fairness based on calibration-related metrics, which is especially relevant in the insurance context, as emphasized by Baumann et al. (2023). Co-Authors: Arthur Charpentier (Université du Québec à Montréal), Ewen Gallic (Aix Marseille Université), Emmanuel Flachaire (Aix Marseille Université), François Hu (Milliman France)