Learning shallow neural networks in high dimensions: SGD dynamics and scaling laws

Add to your list(s) Download to your calendar using vCal

• Denny Wu, Faculty Fellow at the Center for Data Science, New York University and the Flatiron Institute.
• Thursday 20 November 2025, 14:00-15:00
• MR14, Centre for Mathematical Sciences.

If you have a question about this talk, please contact Fernando Ruiz Mazo.

Abstract: We study the sample and time complexity of online stochastic gradient descent (SGD) in learning a two-layer neural network with M orthogonal neurons on isotropic Gaussian data. We focus on the challenging “extensive-width” regime M≫1 and allow for large condition number in the second-layer parameters, covering the power-law scaling a_m= m^{-β} as a special case. We characterize the SGD dynamics for the training of a student two-layer neural network and identify sharp transition times for the recovery of each signal direction. In the power-law setting, our analysis entails that while the learning of individual teacher neurons exhibits abrupt phase transitions, the juxtaposition of emergent learning curves at different timescales results in a smooth scaling law in the cumulative objective.

This talk is co-hosted by the Computer Laboratory AI Research Group and the Informed-AI Hub.

This talk is part of the Machine learning theory series.

This talk is included in these lists:

• All CMS events
• All Talks (aka the CURE list)
• Artificial Intelligence Research Group Talks (Computer Laboratory)
• CMS Events
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• DPMMS info aggregator
• Department of Computer Science and Technology talks and seminars
• Guy Emerson's list
• Hanchen DaDaDash
• Interested Talks
• MR14, Centre for Mathematical Sciences
• Machine learning theory
• Martin's interesting talks
• PhD related
• School of Technology
• Speech Seminars
• Trust & Technology Initiative - interesting events
• bld31
• ml@cl-math
• ndk22's list
• ob366-ai4er
• rp587
• yk373's list
• yk449

Note that ex-directory lists are not shown.