COOKIES: By using this website you agree that we can place Google Analytics Cookies on your device for performance monitoring. |
University of Cambridge > Talks.cam > Cambridge Ellis Unit > The Cambridge ELLIS Unit Seminar Series - Gabriel Synnaeve
The Cambridge ELLIS Unit Seminar Series - Gabriel SynnaeveAdd to your list(s) Download to your calendar using vCal
If you have a question about this talk, please contact . The Cambridge ELLIS Unit Seminar Series holds talks by leading researchers in the area of machine learning and AI. Our next speaker for July 2024 will be Gabriel Synnaeve. Details of his talk can be found below. Title: “Multi-token Prediction and Exploring LM Losses” Abstarct: Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes, and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 more problems on HumanEval and 17 more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes. https://cam-ac-uk.zoom.us/j/85608707597?pwd=Ga0BEodRBJpHkKwvKNdCjabGhBNUYU.1 This talk is part of the Cambridge Ellis Unit series. This talk is included in these lists:
Note that ex-directory lists are not shown. |
Other listsADF: Amsterdam Density Functional, Concepts and Applications Internet Service Providers CPERG: Cambridge Peace and Education Research GroupOther talksWill there be a warning for the next pandemic? Identifying viral targets of concern Bits with Soul Statistical Inference for the Rough Homogenization Limit of Multiscale Fractional Ornstein-Uhlenbeck Processes Rothschild Public Lecture: On Archimedes' Principle TBA Physical Networks Become What They Learn |