University of Cambridge > Talks.cam > Language Technology Lab Seminars > Unlikelihood-training and Back-training for robust natural language understanding

  Browse A-Z

Log in

Cambridge users (raven)details Other usersdetails No account?details

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

Unlikelihood-training and Back-training for robust natural language understanding

Add to your list(s) Download to your calendar using vCal

If you have a question about this talk, please contact Marinela Parovic.

Language models are known to be good at generalization and memorization. These abilities mean that a language model can be directly be used as a knowledge base, e.g., a language model could easily fill the blank in the sentences “The capital of Canada is BLANK ” and “BLANK is the capital of Canada;” with Ottawa, even if these exact syntactic constructions are never seen during training, a task that requires both generalization and memorization. But we also observe that complex phenomena such as negation are commonly ignored by language models, e.g., the model would still predict Ottawa as the answer to “The capital of Canada is not BLANK ”. I will introduce a new training procedure and objective called “unlikelihood training with reference” in order to build language models that understand negation.

In the second part of the talk, I will show that pretrain and fine-tune paradigm breaks in an out-of-distribution setting. For example, question answering and generation models trained on Natural Questions do not generalize to other domains such as education or bio-medical. I will introduce a new technique called back-training that exploits unsupervised data in the target domains much more efficiently than self-training.

This talk is part of the Language Technology Lab Seminars series.

Tell a friend about this talk:

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

J Other medicinal chemistry symposia Exam Passing Material

Other talks

Past ice sheet evolution: West Antarctica during warm climate intervals Local yield stress in model amorphous solids Understanding human adaptation from modern and ancient DNA sequences Some lessons one philosopher drew from thinking about wanting and liking Jamie: "A biomechanical switch regulates the transition towards homeostasis in oesophageal epithelium" Ben: "Integrating pattern formation and cell movements in our understanding of morphogenesis" Material production in a net zero system: An overview
 

© 2006-2024 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity