The potential of synthetic data for more informative evaluation in Visual Question Answering

Add to your list(s) Download to your calendar using vCal

• Alexander Kuhnle, NLIP, University of Cambridge
• Friday 18 May 2018, 12:00-13:00
• FW26, Computer Laboratory.

If you have a question about this talk, please contact Andrew Caines.

Visual Question Answering (VQA) combines language and scene understanding in a straightforward task. However, the popular VQA Dataset exhibits characteristics which make it easy for even a simple system to achieve surprisingly good performance. At the same time, sophisticated model improvements are often barely, if at all, reflected in corresponding performance gains. I will discuss various reasons for that, and why this type of monolithic real-world dataset as sole benchmark thus might be a dead end for VQA . Various synthetic abstract VQA datasets have recently been published, to help overcome this problem. I will introduce “ShapeWorld”, our framework for automatically generating abstract multimodal data, discuss some of the design choices of the generation system, and contrast it with related synthetic datasets. Central to our system is an evaluation methodology akin to “unit-testing”, in contrast to real-world datasets which serve more as application benchmarks. Finally, I will present some experimental results, focusing on quantifier statements, which illustrate this approach and how it enables more targeted and detailed analysis of deep learning models.

This talk is part of the NLIP Seminar Series series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge talks
• Chris Davis' list
• Computer Education Research
• Computing Education Research
• Department of Computer Science and Technology talks and seminars
• FW26, Computer Laboratory
• Graduate-Seminars
• Guy Emerson's list
• Interested Talks
• Language Sciences for Graduate Students
• NLIP Seminar Series
• PMRFPS's
• Simon Baker's List
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.