University of Cambridge > Talks.cam > Language Technology Lab Seminars > "What it can create, it may not understand" Studying the Limits of Transformers.

  Browse A-Z

Log in

Cambridge users (raven)details Other usersdetails No account?details

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

"What it can create, it may not understand" Studying the Limits of Transformers.

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Panagiotis Fytas.

Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. They only take seconds to produce outputs that would challenge or exceed the capabilities even of expert humans. Yet, these models simultaneously show failures on surprisingly trivial problems. This presents us with an apparent paradox: how do we reconcile seemingly superhuman capabilities with the persistence of errors that few humans would make? Are these errors incidental, or do they signal more substantial limitations? In an attempt to demystify Transformers, in this talk, I will discuss the limits of LLMs across three different compositional tasks. Our findings show that although LLMs can outperform humans in generation, they consistently fall short of human capabilities in measures of understanding, showing weaker correlation between generation and understanding performance, and more brittleness to adversarial inputs. We further show that transformers can often solve multi-step compositional problems by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. Overall, our findings support the hypothesis that models’ generative capability may not be contingent upon understanding capability, and call for caution in interpreting artificial intelligence by analogy to human intelligence.

This talk is part of the Language Technology Lab Seminars series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Cambridge University Enterprise Network Exam Practice Questions PDF wowtehcub

Other talks

Does phase separation explain spatial patterns in ecosystems? Statistics Clinic Summer 2024 I Chirality and Topology Stochastic approximation with heavy tailed noise About small jumps of Lévy processes: approximations and estimation A tour of Erdős problems
 

© 2006-2024 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity