University of Cambridge > Talks.cam > Computational Neuroscience > In-Context Learning

  Browse A-Z

Log in

Cambridge users (raven)details Other usersdetails No account?details

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

In-Context Learning

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact .

In-context learning is an emergent capability of large language models (LLMs) trained via next-token prediction. It refers to the LLMs’ ability to learn new tasks and associations (rules, patterns, functions, etc), without changes in their weights, based on (often few) examples provided in their active context window. We will mention some examples of in-context learning of both natural language and numerical tasks by LLMs, and quickly review work on their mechanistic interpretability. In particular, we will present a study linking ICL capabilities of transformer LLMs to the emergence of so-called induction heads during their (pre)training. We will then present a paper which reveals striking parallels between induction heads in LLMs and the Contextual Maintenance and Retrieval (CMR) model of human episodic memory. Both exhibit similar behavioural patterns (temporal contiguity and forward asymmetry), converge on nearly identical parameter values, and use functionally equivalent computational mechanisms. This convergence between artificial and biological systems offers valuable insights into both LLM interpretability and the computational principles underlying sequential memory processing in humans.

Papers: https://transformer-circuits.pub/2021/framework/index.html https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://proceedings.neurips.cc/paper_files/paper/2024/file/0ba385c3ea3bb417ac6d6a33e24411bc-Paper-Conference.pdf

This talk is part of the Computational Neuroscience series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Cambridge Network Sales & Marketing SIG Earthing Spirituality Cambridge Initiative for Musculoskeletal Tissue Engineering

Other talks

Maxwell’s tape measure Prof. Jonathan Wilson Yewdell, Senior Investigator Cellular Biology and Viral Immunology Section, NIAID/DIR Is There Hope for the Climate? Optimization of deep ocean meridional circulation by turbulent mixing QBS TBC
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity