Feature learning and normalization layers

Add to your list(s) Download to your calendar using vCal

• Dr Matus Telgarsky, NYU
• Wednesday 13 March 2024, 14:00-15:00
• MR5, CMS Pavilion A.

If you have a question about this talk, please contact Dr Varun Jog.

Abstract: The first half of this talk will describe the feature learning problem in deep learning optimization, its statistical consequences, and an approach to proving general theorems with a heavy reliance on normalization layers, which are common to all modern architectures but typically treated as an analytic nuisance. Theorems will cover two settings: concrete results for shallow networks, and abstract template theorems for general architectures.

The second half will survey proof techniques. The two key ingredients are a careful new mirror descent lemma, derived from the work of Chizat and Bach, and a new characterization of common layer types called lower homogeneity.

Joint work with Danny Son.

Bio: Matus Telgarsky is an assistant professor in the Courant Institute, NYU , specializing in deep learning theory. He was fortunate to receive a PhD at UCSD under Sanjoy Dasgupta. Other highlights include: co-founding, in 2017, the Midwest ML Symposium (MMLS) with Po-Ling Loh; receiving a 2018 NSF CAREER award; and organizing two Simons Institute programs, one on deep learning theory (summer 2019), and one on generalization (fall 2024, again with Po-Ling Loh); having lots of good friends and too many fun things to do.

This talk is part of the Information Theory Seminar series.

This talk is included in these lists:

• All CMS events
• All Talks (aka the CURE list)
• CMS Events
• DPMMS Lists
• DPMMS info aggregator
• DPMMS lists
• Hanchen DaDaDash
• Information Theory Seminar
• Interested Talks
• MR5, CMS Pavilion A
• School of Physical Sciences
• Statistical Laboratory info aggregator
• bld31

Note that ex-directory lists are not shown.