Linear Transformers for Efficient Sequence Modeling

Add to your list(s) Download to your calendar using vCal

• Prof Yoon Kim, MIT
• Thursday 23 January 2025, 15:00-16:00
• https://cam-ac-uk.zoom.us/j/97599459216?pwd=QTRsOWZCOXRTREVnbTJBdXVpOXFvdz09.

If you have a question about this talk, please contact shun shao.

Abstract:

Transformers are still the dominant architecture for language modeling (and generative AI more broadly). The attention mechanism in Transformers is considered core to the architecture and enables accurate sequence modeling at scale. However, attention requires explicitly modeling pairwise interactions amongst all elements of a sequence, and thus its complexity is quadratic in input length. This talk will describe some recent work from our group on efficient architectural alternatives to Transformers for language modeling, in particular linear Transformers, which can be reparameterized as an RNN and thus allow for linear-time constant-memory sequence modeling. We also provide connections between linear Transformers and recent state-space models such as Mamba.

Bio: Yoon Kim is an assistant professor at MIT (EECS/CSAIL). He obtained his PhD in computer science from Harvard University, where he was advised by Alexander Rush. Prof. Kim works on natural language processing and machine learning. Current interests include: - Efficient training and deployment of large-scale models - Understanding the capabilities and limitations of language models - Symbolic mechanisms for controlling and augmenting neural networks

This talk is part of the Language Technology Lab Seminars series.

This talk is included in these lists:

• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• Language Technology Lab Seminars
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• https://cam-ac-uk.zoom.us/j/97599459216?pwd=QTRsOWZCOXRTREVnbTJBdXVpOXFvdz09
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.