A mean-field theory of lazy training in two-layer neural nets: entropic regularization and controlled McKean-Vlasov dynamics

Add to your list(s) Download to your calendar using vCal

• Maxim Raginsky - University of Illinois at Urbana-Champaign
• Tuesday 13 July 2021, 15:00-16:00
• https://us02web.zoom.us/j/4784296471?pwd=SE8vc1BvWldlZnc1YUNwK3Q1dHpodz09.

If you have a question about this talk, please contact Francisco Vargas.

Paper:

Talk is based on this paper.

Abstract:

We consider the problem of universal approximation of functions by two-layer neural nets with random weights that are “nearly Gaussian” in the sense of Kullback-Leibler divergence. This problem is motivated by recent works on lazy training, where the weight updates generated by stochastic gradient descent do not move appreciably from the i.i.d. Gaussian initialization. We first consider the mean-field limit, where the finite population of neurons in the hidden layer is replaced by a continual ensemble, and show that our problem can be phrased as global minimization of a free-energy functional on the space of probability measures over the weights. This functional trades off the L2 approximation risk against the KL divergence with respect to a centered Gaussian prior. We characterize the unique global minimizer and then construct a controlled nonlinear dynamics in the space of probability measures over weights that solves a McKean—Vlasov optimal control problem. This control problem is closely related to the Schrödinger bridge (or entropic optimal transport) problem, and its value is proportional to the minimum of the free energy. Finally, we show that SGD in the lazy training regime (which can be ensured by jointly tuning the variance of the Gaussian prior and the entropic regularization parameter) serves as a greedy approximation to the optimal McKean—Vlasov distributional dynamics and provide quantitative guarantees on the L2 approximation error.

Website: https://maxim.ece.illinois.edu/

Part of ML@CL Seminar Series in topics relevant to machine learning and statistics.

Join Zoom Meeting https://us02web.zoom.us/j/4784296471?pwd=SE8vc1BvWldlZnc1YUNwK3Q1dHpodz09 Meeting ID: 478 429 6471 Passcode: 345261

This talk is part of the ML@CL Seminar Series series.

This talk is included in these lists:

• Hanchen DaDaDash
• ML@CL Seminar Series
• https://us02web.zoom.us/j/4784296471?pwd=SE8vc1BvWldlZnc1YUNwK3Q1dHpodz09

Note that ex-directory lists are not shown.