BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Emergence of heavy tails in homogenised stochastic gradient descen
 t - Martin Keller-ressel (Technische Universität Dresden)
DTSTART:20240425T141500Z
DTEND:20240425T150000Z
UID:TALK214234@talks.cam.ac.uk
DESCRIPTION:We analyze a continuous diffusion approximation of SGD\, calle
 d homogenized stochastic gradient descent\, show that it behaves asymptoti
 cally heavy-tailed\, and give explicit upper and lower bounds on its tail-
 index. We validate these bounds in numerical experiments and show that the
 y are typically close approximations to the empirical tail-index of SGD it
 erates. In addition\, their explicit form enables us to quantify the inter
 play between optimization hyperparameters and the tail-index. Our results 
 show that also continuous diffusions\, not only L&eacute\;vy-driven SDEs\,
  can accurately represent the emergence of heavy tails in SGD. In addition
 \, our results suggest skew Student-t-distributions\, not alpha-stable dis
 tributions\, as surrogates of parameter distributions under SGD.\n&nbsp\;\
 n&nbsp\;
LOCATION:External
END:VEVENT
END:VCALENDAR