Compositionality decomposed: how do neural networks generalise?

Add to your list(s) Download to your calendar using vCal

• Dieuwke Hupkes, Facebook AI Research
• Thursday 11 February 2021, 13:00-14:00
• Remote.

If you have a question about this talk, please contact Agnieszka Slowik.

In the past years, neural networks have taken over the state of the art in many fields in natural language processing. Despite their successes, such models are frequently argued to not be capable of modelling the types of structures required to adequately model natural language. In particular, deep neural networks are claimed to be relying largely on statistical patterns in the data instead of inferring compositional solutions. Since it is largely unknown what types of composition functions neural network models learn and how they achieve their successes on NLP tasks, it is difficult to refute or proof such claims. As a consequence, even if an experiment convincingly shows that an architecture is not able to perform a particular compositional generalisation, it is often unclear what this means, and how this result relates to linguistic theories about compositionality. In this talk, I present a series of experiments concerning artificial neural networks that aim to take apart different aspects of compositionality that are grounded in linguistic and psycholinguistic literature about this topic. In particular, I focus on the extent to which different architectures are systematic, productive, under what conditions they support substitution of meanings, whether their representations are locally consistent and whether they overgeneralise when faced with exceptions to rules. With this study, we aim to shine some light on the strengths and weaknesses of three different deep learning architectures that are popular for sequential tasks: LST Ms, Convolution-based models and Transformers, as well as aspire to reconnect connectionism and symbolism in the discussion about compositionality and pave the way for modelling approaches that better integrate the two.

This talk is part of the Women@CL Events series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge talks
• Department of Computer Science and Technology talks and seminars
• Guy Emerson's list
• Interested Talks
• Remote
• School of Technology
• Trust & Technology Initiative - interesting events
• Women@CL Events
• bld31
• women@CL all

Note that ex-directory lists are not shown.