Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems using Item Response Theory

Add to your list(s) Download to your calendar using vCal

• Dr. Michal Konsinski
• Tuesday 29 April 2014, 16:30-17:30
• Seminar Room, Department of Psychology, Downing Site, Cambridge.

If you have a question about this talk, please contact Chan Yin Wah Fiona.

Crowd IQ: Weighting Votes in Crowdsourcing and Multi-Agent Systems using Item Response Theory

This work introduces a Weighted Majority Voting (WMV) approach for boosting the performance of crowds and multi-agent systems. Our approach relies on the estimation of performance of individual agents based on their ability modelled using Item Response Theory (IRT), a core theory in modern psychological assessment. We provide a brief introduction to IRT and derive the formula defining the weight of individual agents based on their ability and parameters of the task. Using simulated and empirical samples, we show that WMV (1) offers a significant boost in performance across tasks and crowds of different size, (2) is not substantially affected by not knowing the true task parameters-a common scenario in the real-life setting, (3) significantly increases the chances of the crowd to outperform their smartest member, and (4) boosts the marginal contributions of additional agents. Also, using the example of general intelligence, we show that an agent’s ability can be estimated with appropriate accuracy using a relatively short screening test. We conclude with a discussion of the advantages of WMV and suggest that implementing ability- rather than performance-based metrics can offer great advantages in the context of crowdsourcing and multi-agent platforms.

This talk is part of the Cambridge Psychometrics Centre Seminars series.

This talk is included in these lists:

• Biology
• Cambridge Neuroscience Seminars
• Cambridge Psychometrics Centre Seminars
• Cambridge talks
• Chris Davis' list
• Department of Psychiatry talks stream
• Featured lists
• Life Science
• Life Sciences
• Neuroscience
• Neuroscience Seminars
• Neuroscience Seminars
• Psychology talks and events
• Seminar Room, Department of Psychology, Downing Site, Cambridge
• Stem Cells & Regenerative Medicine
• Yishu's list
• dh539

Note that ex-directory lists are not shown.