Evaluating Deep Generative Models on Out-of-Distribution Inputs

Add to your list(s) Download to your calendar using vCal

• Eric Nalisnick (University of Cambridge)
• Friday 06 March 2020, 12:00-13:00
• FW26, Computer Laboratory.

If you have a question about this talk, please contact James Thorne.

Generative models are widely believed to be more robust to out-of-training-distribution inputs than conditional (i.e. predictive) models. In this talk, I challenge this assumption. We find that the density learned by flow-based models, VAEs, and PixelCNNs cannot distinguish images of common objects such as dogs, trucks, and horses from those of house numbers, assigning a higher likelihood to the latter when the model is trained on the former. We posit that this phenomenon is caused by a mismatch between the model’s typical set and its areas of high probability density. In-distribution inputs should reside in the former but not necessarily in the latter. To determine whether or not inputs reside in the typical set, we propose a computationally efficient hypothesis test using the empirical distribution of model likelihoods. Experiments show that this test succeeds in detecting out-of-distribution inputs in many cases in which previously proposed threshold-based techniques fail.

This talk is part of the NLIP Seminar Series series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Department of Computer Science and Technology talks and seminars
• FW26, Computer Laboratory
• Graduate-Seminars
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• NLIP Seminar Series
• PMRFPS's
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.