University of Cambridge > > CL-CompBio > Understanding the loss landscapes of large neural networks: scaling, generalization, and robustness

Understanding the loss landscapes of large neural networks: scaling, generalization, and robustness

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Pietro Lio.

Large deep neural networks trained with gradient descent have been extremely successful at learning solutions to a broad suite of difficult problems across a wide range of domains. Despite their tremendous success, we still do not have a detailed, predictive understanding of how they work and what makes them so effective. In this talk, I will describe recent efforts to understand the structure of deep neural network loss landscapes and how gradient descent navigates them during training. In particular, I will discuss a phenomenological approach to modeling their large-scale structure using high-dimensional geometry [1], the role of their nonlinear nature in the early phases of training [2], its effects on ensembling, calibration, and approximate Bayesian techniques [3], and the questions of model scaling, multi-modality, pre-training and their connections to out-of-distribution robustness and generalization [4].

[1] Stanislav Fort, and Stanislaw Jastrzebski. “Large Scale Structure of Neural Network Loss Landscapes.” NeurIPS 2019. arXiv 1906.04724

[2] Stanislav Fort et al. “Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel”. NeurIPS 2020. arXiv 2010.15110

[3] Stanislav Fort, Huiyi Hu, Balaji Lakshminarayanan. “Deep Ensembles: A Loss Landscape Perspective.” arXiv 1912.02757

[4] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the Limits of Out-of-Distribution Detection. NeurIPS 2021. arXiv 2106.03004

This talk is part of the CL-CompBio series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.


© 2006-2021, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity