Virtual Seminar: 'Adapting Real-World Experimentation To Balance Enhancement of User Experiences with Statistically Robust Scientific Discovery'

Add to your list(s) Download to your calendar using vCal

• Professor Joseph Jay Williams, University of Toronto
• Thursday 13 May 2021, 14:00-15:00
• Virtual Seminar .

If you have a question about this talk, please contact Alison Quenault.

If you would like to join this virtual seminar, please email alison.quenault@mrc-bsu.cam.ac.uk for more information.

How can we transform the everyday technology people use into intelligent, self-improving systems? For example, how can we perpetually enhance text messages for managing stress, or personalize explanations in online courses? Our work explores the use of randomized adaptive experiments that test alternative actions (e.g. text messages, explanations), aiming to gain greater statistical confidence about the value of actions, in tandem with rapidly using this data to give better actions to future users.

To help characterize the problems that arise in statistical analysis of data collected while trading off exploration and exploitation, we present a real-world case study of applying the multi-armed bandit algorithm TS (Thompson Sampling) to adaptive experiments. TS aims to assign people to actions in proportion to the probability those actions are optimal. We present empirical results on how the reliability of statistical analysis is impacted by Thompson Sampling, compared to a traditional experiment using uniform random assignment. This helps characterize a substantial problem to be solved – using a reward maximizing algorithm can cause substantial issues in statistical analysis of the data. More precisely, an adaptive algorithm can increase both false positives (believing actions have different effects when they do not) and false negatives (failing to detect differences between actions). We show how statistical analyses can be modified to take into account properties of the algorithm, but that these do not fully address the problem raised.

We therefore introduce an algorithm which assigns a proportion of participants uniformly randomly and the remaining participants via Thompson sampling. The probability that a participant is assigned using Uniform Random (UR) allocation is set to the posterior probability that the difference between two arms is ‘small’ (below a certain threshold), allowing for more UR exploration when there is little or no reward to be gained by exploiting. The resulting data can enable more accurate statistical inferences from hypothesis testing by detecting small effects when they exist (reducing false negatives), and reducing false positives.

The work we present aims to surface the underappreciated complexity of using adaptive experimentation to both enable scientific/statistical discovery and help real-world users The current work takes a first step towards computationally characterizing some of the problems that arise, and what potential solutions might look like, in order to inform and invite multidisciplinary collaboration between researchers in machine learning, statistics, and the social-behavioral sciences.

This talk is part of the MRC Biostatistics Unit Seminars series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• BHRU Annual Lecture 2015
• BHRU Annual Lecture 2016
• Biology
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Immunology
• Cambridge talks
• Cardiovascular Epidemiology Unit Special Seminars
• Chris Davis' list
• Department of Public Health and Primary Care
• Graduate-Seminars
• Interested Talks
• Life Sciences
• Life Sciences
• ME Seminar
• MRC Biostatistics Unit Seminars
• Primary Care
• PublicHealth@Cambridge
• Trust & Technology Initiative - interesting events
• Virtual Seminar
• bld31
• my_list
• ndk22's list
• ob366-ai4er
• other talks
• primary care
• rp587

Note that ex-directory lists are not shown.