Coordinate Descent on the Orthogonal Group for Recurrent Neural Network Training

Add to your list(s) Download to your calendar using vCal

• Estelle Massart (Oxford)
• Wednesday 03 November 2021, 14:00-15:00
• Virtual (Zoom details under abstract).

If you have a question about this talk, please contact Hamza Fawzi.

To address the poor scalability of training algorithms for orthogonal recurrent neural networks, we propose to use a coordinate descent method on the orthogonal group. This algorithm has a cost per iteration that evolves linearly with the number of recurrent states, in contrast with the cubic dependency of typical algorithms such as stochastic Riemannian gradient descent. We numerically show that the Riemannian gradient in recurrent neural network training has an approximately sparse structure. Leveraging this observation, we propose a variant of the proposed algorithm that relies on Gauss-Southwell coordinate selection. Experiments on a benchmark recurrent neural network training problem show that the proposed approach is a very promising step towards the training of orthogonal recurrent neural networks with big architectures.

Join Zoom Meeting https://maths-cam-ac-uk.zoom.us/j/93776043287?pwd=UDIrNDdkeUU1NmFtZXpNUzd6ZjRrdz09 Meeting ID: 937 7604 3287 Passcode: p1Co4skf

This talk is part of the CCIMI Seminars series.

This talk is included in these lists:

• All CMS events
• All Talks (aka the CURE list)
• CCIMI
• CCIMI Seminars
• CMS Events
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge talks
• Chris Davis' list
• DPMMS Lists
• DPMMS info aggregator
• DPMMS lists
• Guy Emerson's list
• Hanchen DaDaDash
• Interested Talks
• School of Physical Sciences
• Statistical Laboratory info aggregator
• Trust & Technology Initiative - interesting events
• Virtual (Zoom details under abstract)
• bld31
• custom
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.