Beyond Calibration of Probabilistic Classifier Outputs

Add to your list(s) Download to your calendar using vCal

• Agathe Fernandes Machado (Université du Québec à Montréal)
• Tuesday 03 June 2025, 12:05-12:25
• Seminar Room 1, Newton Institute.

If you have a question about this talk, please contact nobody.

RCLW02 - Calibrating prediction uncertainty : statistics and machine learning perspectives

The assessment of binary classifier performance traditionally centers on discriminative ability, using metrics such as accuracy. However, these metrics often disregard the model’s inherent uncertainty, an aspect that becomes particularly important in sensitive decision-making domains like healthcare or insurance. For instance, when estimating mortality rates or predicting accident occurrence in an insurance context, since model-predicted scores are often interpreted as event probabilities, having a calibrated model allows for enhancing the reliability of these predictions. While calibration enhances the alignment between predicted scores and actual outcomes, a common misconception is to interpret the resulting scores as true posterior probabilities. Specifically, tree-based classifiers may seem calibrated based on standard calibration metrics and exhibit high predictive performance even if their score distributions do not align with true event distributions. In this study, we analyze how the lack of score heterogeneity in tree-based classifiers can lead to misinterpretation of low calibration errors. Furthermore, since the notion of calibration is often used to assess fairness by comparing errors across groups defined by sensitive attributes such as gender or race, a lack of score heterogeneity within minority groups can conceal underlying disparities. Addressing this issue allows for a more accurate evaluation of fairness based on calibration-related metrics, which is especially relevant in the insurance context, as emphasized by Baumann et al. (2023). Co-Authors: Arthur Charpentier (Université du Québec à Montréal), Ewen Gallic (Aix Marseille Université), Emmanuel Flachaire (Aix Marseille Université), François Hu (Milliman France)

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

• All CMS events
• Featured lists
• INI info aggregator
• Isaac Newton Institute Seminar Series
• School of Physical Sciences
• Seminar Room 1, Newton Institute
• bld31

Note that ex-directory lists are not shown.