University of Cambridge > Talks.cam > Language Technology Lab Seminars > Variational Smoothing in Recurrent Neural Network Language Models

Variational Smoothing in Recurrent Neural Network Language Models

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Qianchu Liu.

In this talk, we present a new theoretical perspective of data noising in recurrent neural network language models (Xie et al., 2017). We show that each variant of data noising is an instance of Bayesian recurrent neural networks with a particular variational distribution (i.e., a mixture of Gaussians whose weights depend on statistics derived from the corpus such as the unigram distribution). We use this insight to propose a more principled method to apply at prediction time and propose natural extensions to data noising under the variational framework. In particular, we propose variational smoothing with tied input and output embedding matrices and an element-wise variational smoothing method. We empirically verify our analysis on two bench-mark language modeling datasets and demonstrate performance improvements over existing data noising methods.

This talk is part of the Language Technology Lab Seminars series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

 

© 2006-2020 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity