BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//talks.cam.ac.uk//v3//EN
BEGIN:VTIMEZONE
TZID:Europe/London
BEGIN:DAYLIGHT
TZOFFSETFROM:+0000
TZOFFSETTO:+0100
TZNAME:BST
DTSTART:19700329T010000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
TZNAME:GMT
DTSTART:19701025T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
CATEGORIES:Applied and Computational Analysis
SUMMARY:On the Training of Infinitely Deep and Wide ResNet
s - Gabriel Peyré (École Normale Supérieure)
DTSTART;TZID=Europe/London:20230525T150000
DTEND;TZID=Europe/London:20230525T160000
UID:TALK193588AThttp://talks.cam.ac.uk
URL:http://talks.cam.ac.uk/talk/index/193588
DESCRIPTION:Overparametrization is a key factor in the absence
of convexity to explain global convergence of gra
dient descent (GD) for neural networks. Beside the
well studied lazy regime\, infinite width (mean f
ield) analysis has been developed for shallow netw
orks\, using convex optimization technics. To brid
ge the gap between the lazy and mean field regimes
\, we study Residual Networks (ResNets) in which t
he residual block has linear parametrization while
still being nonlinear. Such ResNets admit both in
finite depth and width limits\, encoding residual
blocks in a Reproducing Kernel Hilbert Space (RKHS
). In this limit\, we prove a local Polyak-Lojasie
wicz inequality. Thus\, every critical point is a
global minimizer and a local convergence result of
GD holds\, retrieving the lazy regime. In contras
t with other mean-field studies\, it applies to bo
th parametric and non-parametric cases under an ex
pressivity condition on the residuals. Our analysi
s leads to a practical and quantified recipe: star
ting from a universal RKHS\, Random Fourier Featur
es are applied to obtain a finite dimensional para
meterization satisfying with high-probability our
expressivity condition.\nThis is a joint work with
Raphaël Barboni and François-Xavier Vialard.
LOCATION:Centre for Mathematical Sciences\, MR14
CONTACT:Matthew Colbrook
END:VEVENT
END:VCALENDAR