COOKIES: By using this website you agree that we can place Google Analytics Cookies on your device for performance monitoring. |

University of Cambridge > Talks.cam > Isaac Newton Institute Seminar Series > Depth Without the Magic: Inductive Bias of Natural Gradient Descent

## Depth Without the Magic: Inductive Bias of Natural Gradient DescentAdd to your list(s) Download to your calendar using vCal - Anna Kerekes (Centre for Mathematical Sciences)
- Tuesday 07 December 2021, 09:00-10:00
- Seminar Room 2, Newton Institute.
If you have a question about this talk, please contact nobody. MDL - Mathematics of deep learning In gradient descent, changing how we parametrize the model can lead to drastically different optimization trajectories, giving rise to a surprising range of meaningful inductive biases: identifying sparse classifiers or reconstructing low-rank matrices without explicit regularization. This implicit regularization has been hypothesised to be a contributing factor to good generalization in deep learning. However, natural gradient descent is approximately invariant to reparameterization, it always follows the same trajectory and finds the same optimum. The question naturally arises: What happens if we eliminate the role of parameterization, which solution will be found, what new properties occur? We characterize the behaviour of natural gradient flow in deep linear networks for separable classification under logistic loss and deep matrix factorization. Some of our findings extend to nonlinear neural networks with sufficient but finite over-parametrization. We demonstrate that there exist learning problems where natural gradient descent fails to generalize, while gradient descent with the right architecture performs well. This talk is part of the Isaac Newton Institute Seminar Series series. ## This talk is included in these lists:- All CMS events
- Featured lists
- INI info aggregator
- Isaac Newton Institute Seminar Series
- School of Physical Sciences
- Seminar Room 2, Newton Institute
- bld31
Note that ex-directory lists are not shown. |
## Other listsCombined External Astrophysics Talks DAMTP Imagine 2017 Talk by Gary Dymski: The Neoclassical Sink and the Heterodox Spiral: Why the Twin Global Crisis has not Transformed Economics## Other talksExploring the Cold Universe: From Obscured Active Galactic Nuclei, Interstellar to Circumgalactic Media at z ~ 5 - 7 Ab Initio: Traversing the abstraction layers: an introduction to graph programming 'Ethno-Science': Guest Speaker – Esther Jean Langdon | gloknos Research Group Creating Barriers: Migration, Citizenship, and the Politics of Inclusion & Exclusion Welcome Talk Discussion Session |