University of Cambridge > > Machine Learning Reading Group @ CUED > Predicting generalization of ML models.

Predicting generalization of ML models.

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact .

Zoom link available upon request (it is sent out on our mailing list, eng-mlg-rcc [at] Sign up to our mailing list for easier reminders.

Generalization is critical for ML applications. The standard measures of generalization using held-out splits of in-distribution (ID) data, however, tend to overestimate performance in the real-world. Moreover, it is unclear whether in-distribution performance correlates well with generalisation. Evaluating generalisation using performance on a few out-of-distribution (OOD) datasets may also fall short due to selection bias. We will begin with a discussion of large scale experiments that study the behaviour of deep learning models on OOD data. We will then discuss empirically developed generalization measures that map a trained model and training data to test error (in the real-world). These measures depend on model properties such as calibration, spectral complexity, smoothness, sensitivity to augmentations, and performance on in-domain data. We will then conclude with a discussion of theoretical developments on this thread.

Required Reading:

1: Assaying Out-Of-Distribution Generalization in Transfer Learning

Summary: A very large scale study of generalization. They study correlation between OOD performance and multiple model properties. They find that ID accuracy is the best predictor of OOD , but some secondary metrics can provide additional insights.

Sections to read: Abstract, Introduction

2: Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning

Summary: Introduces NeurIPS 2020 competition and summarizes its top-3 solutions. Contains a good motivation for the problem and big picture.

Sections to read: Abstract, Introduction, Section 3 (Solutions) up to 3.1.

This talk is part of the Machine Learning Reading Group @ CUED series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.


© 2006-2023, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity