A Contrastive Framework for Neural Text Generation

Add to your list(s) Download to your calendar using vCal

• Yixuan Su, University of Cambridge
• Thursday 02 June 2022, 11:00-12:00
• Board Room, English Faculty, 9 West Road (Sidgwick Site).

If you have a question about this talk, please contact Marinela Parovic.

Text generation is of great importance to many NLP applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions—-the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive training objective to calibrate the model’s representation space, and (ii) a decoding method—-contrastive search—-to encourage diversity while maintaining coherence in the generated text. Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach outperforms state-of-the-art text generation methods as evaluated by both human and automatic metrics.

This talk is part of the Language Technology Lab Seminars series.

This talk is included in these lists:

• Board Room, English Faculty, 9 West Road (Sidgwick Site)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• Language Technology Lab Seminars
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.