Understanding the loss landscapes of large neural networks: scaling, generalization, and robustness

Add to your list(s) Download to your calendar using vCal

• Stanislav Fort, Stanford University
• Friday 15 October 2021, 16:00-17:00
• Department of Computer Science and technology, Lecture Theatre 1.

If you have a question about this talk, please contact Pietro Lio.

Large deep neural networks trained with gradient descent have been extremely successful at learning solutions to a broad suite of difficult problems across a wide range of domains. Despite their tremendous success, we still do not have a detailed, predictive understanding of how they work and what makes them so effective. In this talk, I will describe recent efforts to understand the structure of deep neural network loss landscapes and how gradient descent navigates them during training. In particular, I will discuss a phenomenological approach to modeling their large-scale structure using high-dimensional geometry [1], the role of their nonlinear nature in the early phases of training [2], its effects on ensembling, calibration, and approximate Bayesian techniques [3], and the questions of model scaling, multi-modality, pre-training and their connections to out-of-distribution robustness and generalization [4].

[1] Stanislav Fort, and Stanislaw Jastrzebski. “Large Scale Structure of Neural Network Loss Landscapes.” NeurIPS 2019. arXiv 1906.04724

[2] Stanislav Fort et al. “Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel”. NeurIPS 2020. arXiv 2010.15110

[3] Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan. “Deep Ensembles: A Loss Landscape Perspective.” arXiv 1912.02757

[4] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the Limits of Out-of-Distribution Detection. NeurIPS 2021. arXiv 2106.03004

This talk is part of the CL-CompBio series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge talks
• Department of Computer Science and Technology talks and seminars
• Department of Computer Science and technology, Lecture Theatre 1
• Interested Talks
• School of Technology
• Trust & Technology Initiative - interesting events
• bld31

Note that ex-directory lists are not shown.