Analysing Memorisation in Classification and Translation through Localisation and Cartography

Add to your list(s) Download to your calendar using vCal

• Verna Dankers (University of Edinburgh)
• Friday 24 January 2025, 12:00-13:00
• Zoom link: https://cam-ac-uk.zoom.us/j/4751389294?pwd=Z2ZOSDk0eG1wZldVWG1GVVhrTzFIZz09.

If you have a question about this talk, please contact Suchir Salhan.

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but which examples require memorisation and where in the millions (or billions) of parameters memorisation occurs are questions that remain largely unanswered. In this talk, I first elaborate on the localisation question by examining memorisation in the context of classification in fine-tuned PLMs, using 12 tasks. Our findings give nuance to the generalisation-first memorisation-second hypothesis dominant in the literature and find memorisation to be a gradual process rather than a localised one.

Secondly, I discuss memorisation from the viewpoint of the data using neural machine translation (NMT) models by putting individual data points on a memorisation-generalisation map.

I illustrate how the data points’ characteristics are predictive of memorisation in NMT and describe the influence that subsets of that map have on NMT systems’ performance.

The talk is based on the following two publications:

Dankers, V., & Titov, I. (2024). Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks. ACL -Findings

Dankers, V., Titov, I., & Hupkes, D. (2023). Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation. EMNLP

This talk is part of the NLIP Seminar Series series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Computer Education Research
• Computing Education Research
• Department of Computer Science and Technology talks and seminars
• Graduate-Seminars
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• NLIP Seminar Series
• PMRFPS's
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• Zoom link: https://cam-ac-uk.zoom.us/j/4751389294?pwd=Z2ZOSDk0eG1wZldVWG1GVVhrTzFIZz09
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.