Abstract

Generalization is critical for ML applications. The standard measures of generalization using held-out splits of in-distribution (ID) data, however, tend to overestimate performance in the real-world. Moreover, it is unclear whether in-distribution performance correlates well with generalisation. Evaluating generalisation using performance on a few out-of-distribution (OOD) datasets may also fall short due to selection bias. We will begin with a discussion of large scale experiments that study the behaviour of deep learning models on OOD data. We will then discuss empirically developed generalization measures that map a trained model and training data to test error (in the real-world). These measures depend on model properties such as calibration, spectral complexity, smoothness, sensitivity to augmentations, and performance on in-domain data. We will then conclude with a discussion of theoretical developments on this thread.

Required Reading:

1: Assaying Out-Of-Distribution Generalization in Transfer Learning https://arxiv.org/pdf/2207.09239.pdf

Summary: A very large scale study of generalization. They study correlation between OOD performance and multiple model properties. They find that ID accuracy is the best predictor of OOD , but some secondary metrics can provide additional insights.

Sections to read: Abstract, Introduction

2: Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning http://proceedings.mlr.press/v133/jiang21a/jiang21a.pdf

Summary: Introduces NeurIPS 2020 competition and summarizes its top-3 solutions. Contains a good motivation for the problem and big picture.

Sections to read: Abstract, Introduction, Section 3 (Solutions) up to 3.1.