Dropout as a Structured Shrinkage Prior

Add to your list(s) Download to your calendar using vCal

• Eric Nalisnick (University of Cambridge)
• Friday 28 February 2020, 17:30-19:00
• Auditorium, Microsoft Research Ltd, 21 Station Road, Cambridge, CB1 2FB.

If you have a question about this talk, please contact Microsoft Research Cambridge Talks Admins.

Dropout regularization of deep neural networks has been a mysterious yet effective tool to prevent overfitting. Explanations for its success range from the prevention of “co-adapted” weights to it being a form of cheap Bayesian inference. We propose a novel framework for understanding multiplicative noise in neural networks, considering continuous distributions as well as Bernoulli noise (i.e. dropout). We show that multiplicative noise induces structured shrinkage priors on a network’s weights. We derive the equivalence through reparametrization properties of scale mixtures and without invoking any approximations. We leverage these insights to propose a novel shrinkage framework for resnets, terming the prior “automatic depth determination” as it is the natural analog of “automatic relevance determination” for network depth.

This talk is part of the AI+Pizza series.

This talk is included in these lists:

• AI+Pizza
• Auditorium, Microsoft Research Ltd, 21 Station Road, Cambridge, CB1 2FB
• bld31

Note that ex-directory lists are not shown.