University of Cambridge > > Isaac Newton Institute Seminar Series > Emergence of heavy tails in homogenised stochastic gradient descent

Emergence of heavy tails in homogenised stochastic gradient descent

Add to your list(s) Download to your calendar using vCal

  • UserMartin Keller-ressel (Technische Universität Dresden)
  • ClockThursday 25 April 2024, 15:15-16:00
  • HouseExternal.

If you have a question about this talk, please contact nobody.

TMLW02 - SGD: stability, momentum acceleration and heavy tails

We analyze a continuous diffusion approximation of SGD , called homogenized stochastic gradient descent, show that it behaves asymptotically heavy-tailed, and give explicit upper and lower bounds on its tail-index. We validate these bounds in numerical experiments and show that they are typically close approximations to the empirical tail-index of SGD iterates. In addition, their explicit form enables us to quantify the interplay between optimization hyperparameters and the tail-index. Our results show that also continuous diffusions, not only Lévy-driven SDEs, can accurately represent the emergence of heavy tails in SGD . In addition, our results suggest skew Student-t-distributions, not alpha-stable distributions, as surrogates of parameter distributions under SGD .    

This talk is part of the Isaac Newton Institute Seminar Series series.

Tell a friend about this talk:

This talk is included in these lists:

This talk is not included in any other list

Note that ex-directory lists are not shown.


© 2006-2024, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity