Gradient-based Hyperparameter Optimisation

Add to your list(s) Download to your calendar using vCal

• Ross Clarke (University of Cambridge)
• Wednesday 11 November 2020, 11:05-12:30
• https://eng-cam.zoom.us/j/86068703738?pwd=YnFleXFQOE1qR1h6Vmtwbno0LzFHdz09.

If you have a question about this talk, please contact Elre Oldewage.

We know the ultimate performance of our machine learning systems depends crucially on our training hyperparameters, motivating work to automate their selection. But traditional methods require many repeated runs and scale poorly to large numbers of hyperparameters (including variable schedules and per-parameter optimiser settings). New gradient-based methods aim to address these issues by updating hyperparameters during training itself, providing a more direct update signal than the black-box models used previously. In this talk, we explore the evolution of these methods and some recent developments, culminating in algorithms which can feasibly optimise millions of hyperparameters in parallel with network weights.

Optional Reading:

Our exposition will not assume any pre-reading. However, the following recent paper closely matches our notation and draws on much of the relevant literature, so would provide useful familiarisation for anybody who wishes:

Jonathan Lorraine, Paul Vicol, David Duvenaud, Optimizing Millions of Hyperparameters by Implicit Differentiation, AISTATS 2020

http://proceedings.mlr.press/v108/lorraine20a.html (avoid the out-of-date ArXiv version)

This talk is part of the Machine Learning Reading Group @ CUED series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge University Engineering Department Talks
• Cambridge talks
• Centre for Smart Infrastructure & Construction
• Chris Davis' list
• Computational Continuum Mechanics Group Seminars
• Featured lists
• Guy Emerson's list
• Hanchen DaDaDash
• Inference Group Journal Clubs
• Inference Group Summary
• Information Engineering Division seminar list
• Interested Talks
• ML
• Machine Learning Reading Group
• Machine Learning Reading Group @ CUED
• Machine Learning Summary
• Quantum Matter Journal Club
• Required lists for MLG
• School of Technology
• Simon Baker's List
• TQS Journal Clubs
• Trust & Technology Initiative - interesting events
• bld31
• custom
• https://eng-cam.zoom.us/j/86068703738?pwd=YnFleXFQOE1qR1h6Vmtwbno0LzFHdz09
• ndk22's list
• ob366-ai4er
• rp587
• yk373's list

Note that ex-directory lists are not shown.