Emergence of heavy tails in homogenised stochastic gradient descent

Add to your list(s) Download to your calendar using vCal

• Martin Keller-ressel (Technische Universität Dresden)
• Thursday 25 April 2024, 15:15-16:00
• External.

If you have a question about this talk, please contact nobody.

TMLW02 - SGD: stability, momentum acceleration and heavy tails

We analyze a continuous diffusion approximation of SGD , called homogenized stochastic gradient descent, show that it behaves asymptotically heavy-tailed, and give explicit upper and lower bounds on its tail-index. We validate these bounds in numerical experiments and show that they are typically close approximations to the empirical tail-index of SGD iterates. In addition, their explicit form enables us to quantify the interplay between optimization hyperparameters and the tail-index. Our results show that also continuous diffusions, not only Lévy-driven SDEs, can accurately represent the emergence of heavy tails in SGD . In addition, our results suggest skew Student-t-distributions, not alpha-stable distributions, as surrogates of parameter distributions under SGD .    

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

This talk is not included in any other list

Note that ex-directory lists are not shown.