Semiparametric Language Models

Add to your list(s) Download to your calendar using vCal

• Dani Yogatama
• Thursday 12 November 2020, 14:00-15:00
• https://teams.microsoft.com/l/meetup-join/19%3ameeting_NjYzNzQyOWUtZWM0MC00YzI5LWEwMzMtOGU0ODM4ZDU1NDcw%40thread.v2/0?context=%7b%22Tid%22%3a%2249a50445-bdfa-4b79-ade3-547b4f3986e9%22%2c%22Oid%22%3a%227c409a60-a41c-43c1-a95a-f00471773d03%22%7d.

If you have a question about this talk, please contact Qianchu Liu.

Machine learning models work well on a dataset given enough training examples, but they often fail to isolate and reuse previously acquired knowledge when the data distribution shifts (e.g., when presented with a new dataset or very long context). In contrast, humans are able to learn incrementally and accumulate persistent knowledge to facilitate faster learning of new skills without forgetting old ones.

In this talk, I will argue that obtaining such an ability for a language model requires significant advances in how to represent, store, and reuse knowledge acquired from textual data. I will present a semiparametric language model framework that separates computation (information processing) in a large parametric neural network and memory storage in a non-parametric component. I will show two instantiations of such a model. First, I will discuss how to use it to allow a language model to continually learn new tasks without forgetting old ones. Second, I will present a language model architecture that adaptively combines local context and global context to make more accurate predictions.

This talk is part of the Language Technology Lab Seminars series.

This talk is included in these lists:

• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• Language Technology Lab Seminars
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• https://teams.microsoft.com/l/meetup-join/19%3ameeting_NjYzNzQyOWUtZWM0MC00YzI5LWEwMzMtOGU0ODM4ZDU1NDcw%40thread.v2/0?context=%7b%22Tid%22%3a%2249a50445-bdfa-4b79-ade3-547b4f3986e9%22%2c%22Oid%22%3a%227c409a60-a41c-43c1-a95a-f00471773d03%22%7d
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.