University of Cambridge > Talks.cam > CUED Speech Group Seminars > Should Ensemble Members Be Calibrated?

Should Ensemble Members Be Calibrated?

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Dr Kate Knill.

Seminar on zoom

Abstract: Underlying the use of statistical approaches for a wide range of applications is the assumption that the probabilities obtained from a statistical model are representative of the “true” probability that event, or outcome, will occur. Unfortunately, for modern deep neural networks this is not the case, they are often observed to be poorly calibrated. Additionally, these deep learning approaches make use of large numbers of model parameters, motivating the use of Bayesian, or ensemble approximation, approaches to handle issues with parameter estimation. This paper explores the application of calibration schemes to deep ensembles from a theoretical perspective. The theoretical requirements for calibration, and associated calibration criteria, are first described. It is shown that well calibrated ensemble members do not necessarily yield a well calibrated ensemble prediction. Furthermore if the ensemble prediction is well calibrated then its performance cannot exceed that of the average performance of the calibrated ensemble members. Empirical results on CIFAR -100 are used to support these theoretical developments. Additionally the relationships between ensemble calibration for classification and regression are discussed.

Bio: Xixin Wu is a Research Associate in the Speech Group of the Machine Intelligence Laboratory, Engineering Department of Cambridge University. He obtained his PhD degree from The Chinese University of Hong Kong. His research interests include speech recognition and synthesis, speaker verification and neural network uncertainty. Xixin is a member of IEEE and ISCA .

This talk is part of the CUED Speech Group Seminars series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

© 2006-2021 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity