BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Virtual Seminar: 'Adapting Real-World Experimentation To Balance E
 nhancement of User Experiences with Statistically Robust Scientific Discov
 ery' - Professor Joseph Jay Williams\, University of Toronto 
DTSTART:20210513T130000Z
DTEND:20210513T140000Z
UID:TALK160390@talks.cam.ac.uk
CONTACT:Alison Quenault
DESCRIPTION:How can we transform the everyday technology people use into i
 ntelligent\, self-improving systems? For example\, how can we perpetually 
 enhance text messages for managing stress\, or personalize explanations in
  online courses? Our work explores the use of randomized adaptive experime
 nts that test alternative actions (e.g. text messages\, explanations)\, ai
 ming to gain greater statistical confidence about the value of actions\, i
 n tandem with rapidly using this data to give better actions to future use
 rs. \n \nTo help characterize the problems that arise in statistical analy
 sis of data collected while trading off exploration and exploitation\, we 
 present a real-world case study of applying the multi-armed bandit algorit
 hm TS (Thompson Sampling) to adaptive experiments. TS aims to assign peopl
 e to actions in proportion to the probability those actions are optimal. W
 e present empirical results on how the reliability of statistical analysis
  is impacted by Thompson Sampling\, compared to a traditional experiment u
 sing uniform random assignment. This helps characterize a substantial prob
 lem to be solved – using a reward maximizing algorithm can cause substan
 tial issues in statistical analysis of the data. More precisely\, an adapt
 ive algorithm can increase both false positives (believing actions have di
 fferent effects when they do not) and false negatives (failing to detect d
 ifferences between actions). We show how statistical analyses can be modif
 ied to take into account properties of the algorithm\, but that these do n
 ot fully address the problem raised.\n \nWe therefore introduce an algorit
 hm which assigns a proportion of participants uniformly randomly and the r
 emaining participants via Thompson sampling. The probability that a partic
 ipant is assigned using Uniform Random (UR) allocation is set to the poste
 rior probability that the difference between two arms is 'small' (below a 
 certain threshold)\, allowing for more UR exploration when there is little
  or no reward to be gained by exploiting. The resulting data can enable mo
 re accurate statistical inferences from hypothesis testing by detecting sm
 all effects when they exist (reducing false negatives)\, and reducing fals
 e positives.\n \nThe work we present aims to surface the underappreciated 
 complexity of using adaptive experimentation to both enable scientific/sta
 tistical discovery and help real-world users The current work takes a firs
 t step towards computationally characterizing some of the problems that ar
 ise\, and what potential solutions might look like\, in order to inform an
 d invite multidisciplinary collaboration between researchers in machine le
 arning\, statistics\, and the social-behavioral sciences.\n
LOCATION:Virtual Seminar 
END:VEVENT
END:VCALENDAR