Making and breaking tokenizers

Add to your list(s) Download to your calendar using vCal

• Sander Land (Writer)
• Friday 17 October 2025, 12:00-13:00
• SS03 Hybrid (In-Person + Online). Google Meet: https://meet.google.com/yeu-pqce-rsn .

If you have a question about this talk, please contact Suchir Salhan.

Despite massive investments in training large language models, tokenizers remain a critical but often neglected component with weaknesses that can cause wild hallucinations, bypass safety guardrails, and break downstream applications. This talk will cover:

Our recent research in automatically detecting problematic ‘glitch’ tokens in any model

Fundamental issues with pretokenizers and their design

Novel approaches to encodings and pretokenization that address some of these problems.

Speaker Bio Sander Land is a researcher at Writer, previously working at Cohere. He completed his PhD at the Department of Computer Science, University of Oxford, before undertaking a postdoc at Biomedical Engineering, King’s College London, University of London.

This talk is part of the NLIP Seminar Series series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Computer Education Research
• Computing Education Research
• Department of Computer Science and Technology talks and seminars
• Graduate-Seminars
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• Martin's interesting talks
• NLIP Seminar Series
• PMRFPS's
• SS03 Hybrid (In-Person + Online). Google Meet: https://meet.google.com/yeu-pqce-rsn
• SS03 Hybrid (In-Person + Online). Google Meet: https://meet.google.com/yeu-pqce-rsn
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587
• yk449

Note that ex-directory lists are not shown.