Accelerating Variance-Reduced Stochastic Gradient Methods

Add to your list(s) Download to your calendar using vCal

• Derek Driggs (University of Cambridge)
• Monday 04 November 2019, 12:00-12:30
• Seminar Room 2, Gatehouse Newton Institute.

If you have a question about this talk, please contact Matthew Colbrook.

Variance reduction is a crucial tool for improving the slow convergence of stochastic gradient descent. Only a few variance-reduced methods, however, have yet been shown to directly benefit from Nesterov’s acceleration techniques to match the convergence rates of accelerated gradient methods. In this work, we show develop a universal acceleration framework that allows all popular variance-reduced methods to achieve accelerated convergence rates. The constants appearing in these rates, including their dependence on the dimension n, scale with the mean-squared-error and bias of the gradient estimator. In a series of numerical experiments, we demonstrate that versions of the popular gradient estimators SAGA , SVRG, SARAH , and SARGE using our framework significantly outperform non-accelerated versions.

This talk is part of the Cambridge-Imperial Computational PhD Seminar series.

This talk is included in these lists:

• Seminar Room 2, Gatehouse Newton Institute

Note that ex-directory lists are not shown.