Estimating and Calibrating Uncertainty in LLMs

Add to your list(s) Download to your calendar using vCal

• Desi Ivanova (University of Oxford)
• Thursday 28 August 2025, 14:00-14:30
• Seminar Room 1, Newton Institute.

If you have a question about this talk, please contact nobody.

RCLW04 - Early Career Pioneers in Uncertainty Quantification and AI for Science

Large Language Models (LLMs) often sound certain when they are wrong. With rapid adoption in critical sectors like healthcare, scientific discovery and research automation, where the consequences of errors can be substantial, reliable Uncertainty Quantification (UQ) becomes crucial for safe and trustworthy deployment. In this talk, we will first review the current landscape of UQ methods for LLMs and discuss the trade-offs between calibration, discrimination, and compute. We’ll then look at how generation temperature (the scalar that controls generative diversity) affects calibration at a semantic level. We’ll introduce a semantic calibration framework and show that simple post-hoc temperature scaling markedly improves meaning-level calibration and selective prediction across open- and closed-book question-answering tasks.

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

• All CMS events
• Featured lists
• INI info aggregator
• Isaac Newton Institute Seminar Series
• School of Physical Sciences
• Seminar Room 1, Newton Institute
• bld31

Note that ex-directory lists are not shown.