Information-Theoretic Generalization Bounds for Stochastic Gradient Descent

Add to your list(s) Download to your calendar using vCal

• Gergely Neu
• Tuesday 20 April 2021, 15:00-16:00
• https://cl-cam-ac-uk.zoom.us/j/97019034279?pwd=UWNGQ2lhSzZGK09TdXVkSHBGRUFZdz09.

If you have a question about this talk, please contact .

We study the generalization properties of the popular stochastic gradient descent method for optimizing general non-convex loss functions. Our main contribution is providing upper bounds on the generalization error that depend on local statistics of the stochastic gradients evaluated along the path of iterates calculated by SGD . The key factors our bounds depend on are the variance of the gradients (with respect to the data distribution) and the local smoothness of the objective function along the SGD path, and the sensitivity of the loss function to perturbations to the final output. Our key technical tool is combining the information-theoretic generalization bounds previously used for analyzing randomized variants of SGD with a perturbation analysis of the iterates.

This talk is part of the ML@CL Seminar Series series.

This talk is included in these lists:

• Hanchen DaDaDash
• ML@CL Seminar Series
• https://cl-cam-ac-uk.zoom.us/j/97019034279?pwd=UWNGQ2lhSzZGK09TdXVkSHBGRUFZdz09

Note that ex-directory lists are not shown.