University of Cambridge > Talks.cam > Isaac Newton Institute Seminar Series > Emergence of heavy tails in homogenised stochastic gradient descent

  Browse A-Z

Log in

University Account External (via Google)

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹ️ Information

Emergence of heavy tails in homogenised stochastic gradient descent

Download to your calendar using vCal

  • UserMartin Keller-ressel (Technische Universität Dresden)
  • ClockThursday 25 April 2024, 15:15-16:00
  • HouseExternal.

If you have a question about this talk, please contact nobody.

TMLW02 - SGD: stability, momentum acceleration and heavy tails

We analyze a continuous diffusion approximation of SGD , called homogenized stochastic gradient descent, show that it behaves asymptotically heavy-tailed, and give explicit upper and lower bounds on its tail-index. We validate these bounds in numerical experiments and show that they are typically close approximations to the empirical tail-index of SGD iterates. In addition, their explicit form enables us to quantify the interplay between optimization hyperparameters and the tail-index. Our results show that also continuous diffusions, not only Lévy-driven SDEs, can accurately represent the emergence of heavy tails in SGD . In addition, our results suggest skew Student-t-distributions, not alpha-stable distributions, as surrogates of parameter distributions under SGD .    

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Room GS15 at the William Gates Building and on Zoom: https://cl-cam-ac-uk.zoom.us/j/4361570789?pwd=Nkl2T3ZLaTZwRm05bzRTOUUxY3Q4QT09&from=addon Webinar & FW11, Computer Laboratory, William Gates Building. Seminar Room (Level 2), Dept of Pharmacology .

Other talks

Group Work Science & the Public in Action: Can and should climate scientists be activists? Group Work Title to be confirmed Differences between Robin and Neumann eigenvalues Online Causal Inference Seminar
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity