Abstract

Variational mean field approximations tend to struggle with contemporary overparameterised deep neural networks. Where a Bayesian treatment is usually associated with high-quality predictions and uncertainties, the practical reality has been the opposite, with unstable training, poor predictive power, and subpar calibration. Building upon recent work on reparameterisations of neural networks, we propose a simple variational family that considers two independent linear subspaces of the parameter space. These represent functional changes inside and outside the support of training data. This allows us to build a fully-correlated approximate posterior reflecting the overparameterisation that tunes easy-to-interpret hyperparameters. We develop scalable numerical routines that maximize the associated evidence lower bound (ELBO) and sample from the approximate posterior. Our results show that approximate Bayesian inference applied to deep neural networks is far from a lost cause when constructing inference mechanisms that reflect the geometry of reparametrisations.

Bio: Samuel is a postdoc at Søren Hauberg’s group at the Technical University of Denmark and an affiliated researcher at the Pioneer Centre for AI. Samuel received his PhD in Computer Science from the University of Campinas, Brazil and an MSc in Applied Mathematics and Computer Science from the University of São Paulo, Brazil. His research interests include representation learning and uncertainty quantification, both with a geometric perspective.

This talk is co-hosted with the Machine Learning Theory Group in the Maths Department and the Informed-AI Hub.