The Transformer (OOD) House of Cards

Add to your list(s) Download to your calendar using vCal

• Petar Veličković (Google Deepmind)
• Thursday 10 October 2024, 14:00-15:00
• Maxwell Centre.

If you have a question about this talk, please contact Sri Aitken.

The Transformer architecture has certainly been the landmark deep learning model in recent years, enabling seamless integration of information across many different modalities and surprisingly insightful behaviours emerging at scale. However, in spite of the very challenging problems that are now within reach of Transformers, they are also seemingly unable to robustly perform when faced with variations of, comparatively, much simpler problems. We attribute this to shaky foundations: there are certain kinds of computations that are always going to be out of reach of Transformers, no matter how well we train them—and a lot of such computations occur outside of the distribution the model was trained on. In this talk, I will outline some of these cracks in the system we’ve discovered, as well as ideas for the way forward towards building generally intelligent agents of the future.

This talk is part of the Data Intensive Science Seminar Series series.

This talk is included in these lists:

• Cambridge Astronomy Talks
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge talks
• Chris Davis' list
• Combined External Astrophysics Talks DAMTP
• Cosmology, Astrophysics and General Relativity
• Institute of Astronomy Extra Talks
• Institute of Astronomy Talk Lists
• Interested Talks
• Maxwell Centre
• Titel: TBC
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.