Collaborative Pretraining on Evolving Pretraining and Small Manageable Tasks

Add to your list(s) Download to your calendar using vCal

• Leshem Choshen (IBM AI research, Hebrew University of Jerusalem)
• Friday 20 October 2023, 12:00-13:00
• https://cam-ac-uk.zoom.us/j/86071371348?pwd=OVlqdDhZNHlGbzV5RUZrSzM1cUlhUT09#success.

If you have a question about this talk, please contact Richard Diehl Martinez.

Pretraining is monolithic. In this talk, I will discuss a collaborative approach to pertaining, by iterative model merging (originally fusing). We will then discuss making evaluation reliable and efficient, to allow anyone to evaluate. We might mention the BabyLM challenge, of pretraining models with human feasible amount of data as well (If interested in more, contact me, babyLM would be CoNLL’s shared task next year as well).

Leshem Choshen is a postdoctoral researcher at MIT -IBM, aiming to collaboratively pretrain through model recycling, efficient evaluation, and efficient pretraining research (e.g., babyLM). He received the postdoctoral Rothschild and Fulbright fellowship as well as IAAI and Blavatnik best Ph.D. awards. With broad NLP and ML interests, he also worked on Reinforcement Learning, Evaluation and Understanding of how neural networks learn. In parallel, he participated in Project Debater, creating a machine that could hold a formal debate, ending in a Nature cover and live debate.

He is also a dancer and runs tei.ma, a food and science blog (NisuiVeTeima on Instagram, Facebook and Tiktok).

This talk is part of the NLIP Seminar Series series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Department of Computer Science and Technology talks and seminars
• Graduate-Seminars
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• NLIP Seminar Series
• PMRFPS's
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• https://cam-ac-uk.zoom.us/j/86071371348?pwd=OVlqdDhZNHlGbzV5RUZrSzM1cUlhUT09#success
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.