University of Cambridge > Talks.cam > Isaac Newton Institute Seminar Series > Beyond Calibration of Probabilistic Classifier Outputs

Beyond Calibration of Probabilistic Classifier Outputs

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact nobody.

RCLW02 - Calibrating prediction uncertainty : statistics and machine learning perspectives

The assessment of binary classifier performance traditionally centers on discriminative ability, using metrics such as accuracy. However, these metrics often disregard the model’s inherent uncertainty, an aspect that becomes particularly important in sensitive decision-making domains like healthcare or insurance. For instance, when estimating mortality rates or predicting accident occurrence in an insurance context, since model-predicted scores are often interpreted as event probabilities, having a calibrated model allows for enhancing the reliability of these predictions. While calibration enhances the alignment between predicted scores and actual outcomes, a common misconception is to interpret the resulting scores as true posterior probabilities. Specifically, tree-based classifiers may seem calibrated based on standard calibration metrics and exhibit high predictive performance even if their score distributions do not align with true event distributions. In this study, we analyze how the lack of score heterogeneity in tree-based classifiers can lead to misinterpretation of low calibration errors. Furthermore, since the notion of calibration is often used to assess fairness by comparing errors across groups defined by sensitive attributes such as gender or race, a lack of score heterogeneity within minority groups can conceal underlying disparities. Addressing this issue allows for a more accurate evaluation of fairness based on calibration-related metrics, which is especially relevant in the insurance context, as emphasized by Baumann et al. (2023). Co-Authors: Arthur Charpentier (Université du Québec à Montréal), Ewen Gallic (Aix Marseille Université), Emmanuel Flachaire (Aix Marseille Université), François Hu (Milliman France)

This talk is part of the Isaac Newton Institute Seminar Series series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity