Algorithmic stability for heavy-tailed SGD

Add to your list(s) Download to your calendar using vCal

• Lingjiong Zhu (Florida State University)
• Wednesday 24 April 2024, 14:30-15:15
• External.

If you have a question about this talk, please contact nobody.

TMLW02 - SGD: stability, momentum acceleration and heavy tails

Recent studies have shown that heavy tails can emerge in stochastic optimization and that the heaviness of the tails have links to the generalization error. In this study, we establish novel links between the tail behavior and generalization properties of stochastic gradient descent (SGD), through the lens of algorithmic stability. We develop generalization bounds for a general class of objective functions, which includes non-convex functions as well. Our approach is based on developing Wasserstein stability bounds for heavy-tailed SGD , which we then convert to generalization bounds, indicating a non-monotonic relationship between the generalization error and heavy tails. We support our theory with synthetic and real neural network experiments.   

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

This talk is not included in any other list

Note that ex-directory lists are not shown.