Using gradient descent for optimization and learning

Add to your list(s) Download to your calendar using vCal

• Nicolas Le Roux (Microsoft Research)
• Tuesday 26 May 2009, 13:00-15:00
• Engineering Department, CBL Room 438.

If you have a question about this talk, please contact Simon Lacoste-Julien.

In machine learning, to solve a particular task, we often define a cost function which we are trying to minimize over a set of data points, the training set. There has been extensive work on designing efficient optimization techniques to minimize such functions, amongst them BFGS and conjugate gradient descent. However, this only ensures a fast decrease in the training error, whereas we are ultimately interested in minimizing another error: the test error. Reaching a low test error for a particular problem is called learning. As Bottou has shown, good optimizers may prove to be very bad learning methods and vice-versa.

After reviewing the most popular optimization techniques, I shall make a brief summary of Bottou’s conclusions and then present a new gradient descent algorithm which aims at directly addressing this issue of optimizing the test error using only a training set.

Speaker’s bio:

Nicolas received a Master’s Degree in Applied Maths from the Ecole Centrale Paris and one in Maths, Vision and Learning from the ENS Cachan in 2003. From 2004 to 2008, he did a PhD in Montreal under the supervision of Yoshua Bengio, working on designing and optimizing neural networks. Since 2008, he is a postdoc researcher at Microsoft Research Cambridge, working with John Winn on using deep neural networks for vision.

This talk is part of the Machine Learning @ CUED series.

This talk is included in these lists:

• Seminar
• All Talks (aka the CURE list)
• Biology
• CBL important
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge Neuroscience Seminars
• Cambridge talks
• Chris Davis' list
• Creating transparent intact animal organs for high-resolution 3D deep-tissue imaging
• Engineering Department, CBL Room 438
• Featured lists
• Guy Emerson's list
• Hanchen DaDaDash
• Inference Group Summary
• Information Engineering Division seminar list
• Interested Talks
• Joint Machine Learning Seminars
• Life Science
• Life Sciences
• ML
• Machine Learning @ CUED
• Machine Learning Summary
• Neuroscience
• Neuroscience Seminars
• Neuroscience Seminars
• Required lists for MLG
• Simon Baker's List
• Stem Cells & Regenerative Medicine
• Trust & Technology Initiative - interesting events
• bld31
• dh539
• ndk22's list
• ob366-ai4er
• rp587
• yk373's list

Note that ex-directory lists are not shown.