BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Predicting generalization of ML models. - Vihari Piratla and Shrey
 as Padhy
DTSTART:20221116T110000Z
DTEND:20221116T123000Z
UID:TALK192794@talks.cam.ac.uk
CONTACT:86986
DESCRIPTION:Generalization is critical for ML applications. The standard m
 easures of generalization using held-out splits of in-distribution (ID) da
 ta\, however\, tend to overestimate performance in the real-world. Moreove
 r\, it is unclear whether in-distribution performance correlates well with
  generalisation. Evaluating generalisation using performance on a few out-
 of-distribution (OOD) datasets may also fall short due to selection bias. 
 We will begin with a discussion of large scale experiments that study the 
 behaviour of deep learning models on OOD data. We will then discuss empiri
 cally developed generalization measures that map a trained model and train
 ing data to test error (in the real-world). These measures depend on model
  properties such as calibration\, spectral complexity\, smoothness\, sensi
 tivity to augmentations\, and performance on in-domain data. We will then 
 conclude with a discussion of theoretical developments on this thread.\n\n
 Required Reading:\n\n1: Assaying Out-Of-Distribution Generalization in Tra
 nsfer Learning https://arxiv.org/pdf/2207.09239.pdf \n\nSummary: A very la
 rge scale study of generalization. They study correlation between OOD perf
 ormance and multiple model properties. They find that ID accuracy is the b
 est predictor of OOD\, but some secondary metrics can provide additional i
 nsights.\n\n*Sections to read:* Abstract\, Introduction\n\n2: Methods and 
 Analysis of The First Competition in Predicting Generalization of Deep Lea
 rning http://proceedings.mlr.press/v133/jiang21a/jiang21a.pdf \n\nSummary:
  Introduces NeurIPS 2020 competition and summarizes its top-3 solutions. C
 ontains a good motivation for the problem and big picture.\n\n*Sections to
  read:* Abstract\, Introduction\, Section 3 (Solutions) up to 3.1.
LOCATION:Cambridge University Engineering Department\, CBL Seminar room BE
 4-38.
END:VEVENT
END:VCALENDAR