Calibration properties of conformal predictive systems

Add to your list(s) Download to your calendar using vCal

• Sam Allen (Karlsruhe Institute of Technology (KIT))
• Wednesday 27 August 2025, 15:30-16:00
• Seminar Room 1, Newton Institute.

If you have a question about this talk, please contact nobody.

RCLW04 - Early Career Pioneers in Uncertainty Quantification and AI for Science

Conformal prediction has gained considerable attention recently since it produces forecasts that have out-of-sample calibration guarantees by construction, under the assumption of exchangeability. In this work, we study the calibration properties of conformal predictive systems, which issue sets of predictive distributions for real-valued outcomes. We demonstrate that conformal predictive systems implicitly exploit prediction methods with in-sample calibration guarantees to construct prediction sets with out-of-sample calibration guarantees. While the calibration guarantees are typically that prediction intervals derived from the predictive distributions have the correct marginal coverage, we show that this line of reasoning can be extended to stronger notions of calibration that are common in statistical forecasting theory. This allows us to take any prediction method that is calibrated in-sample, and conformalize it to obtain a conformal predictive system with out-of-sample calibration guarantees. Using this, we introduce two conformal predictive systems that exhibit stronger calibration properties than existing approaches. The first method corresponds to a binning of the data, while the second leverages isotonic distributional regression (IDR), a non-parametric distributional regression method under order constraints. We study the theoretical properties of these new conformal predictive systems, and compare their performance in a simulation experiment and two case studies. Both approaches are found to outperform existing conformal predictive systems, while conformal IDR additionally provides a natural method for quantifying epistemic uncertainty of the predictions. 

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

• All CMS events
• Featured lists
• INI info aggregator
• Isaac Newton Institute Seminar Series
• School of Physical Sciences
• Seminar Room 1, Newton Institute
• bld31

Note that ex-directory lists are not shown.