A Geometrical Perspective on Deep Neural Networks

Add to your list(s) Download to your calendar using vCal

• Stanislav Fort (Stanford University; formerly Google Research)
• Monday 13 January 2020, 13:00-14:00
• LT2, Computer Laboratory, William Gates Building.

If you have a question about this talk, please contact Ben Day.

Deep neural networks trained with gradient descent have been extremely successful at learning solutions to a broad suite of difficult problems across a wide range of domains such as vision, gameplay, and natural language, many of which had previously been considered to require intelligence. Despite their tremendous success, however, we still do not have a detailed, predictive understanding of how these systems work. In this talk, I will focus on recent insights into the structure of neural network loss landscapes and how they are navigated by gradient descent during training. In particular, I will discuss a phenomenological approach to modelling their large-scale structure [1,2], and its consequences for ensembling, calibration and Bayesian methods in general [3]. In addition, I will make a connection to empirical observations about loss gradients and Hessians [4,5]. I will conclude with an outlook on several interesting open questions in understanding deep networks.

1. Fort, Stanislav, and Adam Scherlis. “The Goldilocks zone: Towards better understanding of neural network loss landscapes.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 2019. arXiv 1807.02581
2. Stanislav Fort, and Stanislaw Jastrzebski. “Large Scale Structure of Neural Network Loss Landscapes.” Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arXiv 1906.04724
3. S Fort, H Hu, B Lakshminarayanan. “Deep Ensembles: A Loss Landscape Perspective.” arXiv 1912.02757
4. Stanislav Fort, Paweł Krzysztof Nowak, Stanislaw Jastrzebski, Srini Narayanan. “Stiffness: A New Perspective on Generalization in Neural Networks.” arXiv 1901.09491
5. Stanislav Fort, Surya Ganguli. “Emergent properties of the local geometry of neural loss landscapes.” arXiv 1910.05929

This talk is part of the Artificial Intelligence Research Group Talks (Computer Laboratory) series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Artificial Intelligence Research Group Talks (Computer Laboratory)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Department of Computer Science and Technology talks and seminars
• Guy Emerson's list
• Hanchen DaDaDash
• Interested Talks
• LT2, Computer Laboratory, William Gates Building
• School of Technology
• Speech Seminars
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587
• yk373's list

Note that ex-directory lists are not shown.