Should Ensemble Members Be Calibrated?

Add to your list(s) Download to your calendar using vCal

• Dr Xixin Wu, Cambridge University Engineering Department
• Tuesday 16 February 2021, 12:00-13:00
• Zoom: https://zoom.us/j/95352633552?pwd=RzJVK2UzOGZyNU5mVHd1Y1VPT2tDUT09.

If you have a question about this talk, please contact Dr Kate Knill.

Seminar on zoom

Abstract: Underlying the use of statistical approaches for a wide range of applications is the assumption that the probabilities obtained from a statistical model are representative of the “true” probability that event, or outcome, will occur. Unfortunately, for modern deep neural networks this is not the case, they are often observed to be poorly calibrated. Additionally, these deep learning approaches make use of large numbers of model parameters, motivating the use of Bayesian, or ensemble approximation, approaches to handle issues with parameter estimation. This paper explores the application of calibration schemes to deep ensembles from a theoretical perspective. The theoretical requirements for calibration, and associated calibration criteria, are first described. It is shown that well calibrated ensemble members do not necessarily yield a well calibrated ensemble prediction. Furthermore if the ensemble prediction is well calibrated then its performance cannot exceed that of the average performance of the calibrated ensemble members. Empirical results on CIFAR -100 are used to support these theoretical developments. Additionally the relationships between ensemble calibration for classification and regression are discussed.

Bio: Xixin Wu is a Research Associate in the Speech Group of the Machine Intelligence Laboratory, Engineering Department of Cambridge University. He obtained his PhD degree from The Chinese University of Hong Kong. His research interests include speech recognition and synthesis, speaker verification and neural network uncertainty. Xixin is a member of IEEE and ISCA .

This talk is part of the CUED Speech Group Seminars series.

This talk is included in these lists:

• CUED Speech Group Seminars
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Guy Emerson's list
• Information Engineering Division seminar list
• PhD related
• Zoom: https://zoom.us/j/95352633552?pwd=RzJVK2UzOGZyNU5mVHd1Y1VPT2tDUT09

Note that ex-directory lists are not shown.