Forward and Inverse models in Human-Computer Interaction: Physical simulation and machine learning for inferring 3D finger pose

Add to your list(s) Download to your calendar using vCal

• Professor Roderick Murray-Smith, University of Glasgow
• Thursday 06 February 2020, 11:00-12:00
• Computer Laboratory, William Gates Building, Room FW26.

If you have a question about this talk, please contact Prof Neil Lawrence.

I will give a brief overview of the research activities in the Information, Data & Analysis Section at the University of Glasgow, touching on some of our recent applications of Machine Learning in optics, nuclear physics & gravitational waves, and our Closed-loop Data Science research project.

I will then outline the role of computational methods in the design of human-computer interaction, and more specifically the role of forward and inverse modelling approaches. Causal, forward models tend to be easier to specify and simulate, but the inverse problem (‘what was the user’s intention’) is what typically needs to be solved in an HCI context. We illustrate the core issues in a case study, where we quantify the accuracy with which single finger 3D position (x,y,z) and pose (pitch and yaw) can be inferred in real time on a mobile device using recent developments in capacitive sensing technology which can sense the finger up to 5cm above a mobile phone screen. We use machine learning to develop data-driven models to infer position, pose and sensor readings, based on three approaches to gathering training data: 1. data generated by robots, 2. data from electrostatic simulators and 3. human-generated data. A forward model is trained on this data using a deep convolutional network. This emulation can accelerate the electrostatic simulation performance with a speedup factor of 2.4 million. We compare forward and inverse model approaches to inference of finger pose and fuse them with a probabilistic filter. This combination of forward and inverse models improves performance over previous inverse-only approaches, giving the most accurate reported results on inferring 3D position and pose with a mobile phone capacitive sensor. I will then give an outlook on how we can improve on this with recently developed variational inference approaches, and discuss the potential of these methods for moving towards a more constructive human model-based pipeline for design of human-computer systems.

Some related publications: Inverse methods via ML & for HCI Francesco Tonolini, Jack Radford, Alex Turpin, Daniele Faccio, Roderick Murray-Smith, Variational Inference for Computational Imaging Inverse Problems, Jan 2020 https://arxiv.org/abs/1904.06264

R. Murray-Smith, Stratified, computational interaction via machine learning, The Eighteenth Yale Workshop on Adaptive and Learning Systems, Ed. K. Narendra, Yale, June 21-23rd, 2017. http://www.dcs.gla.ac.uk/_{rod/publications/Mur17.pdf}

J. Müller, A. Oulasvirta, R. Murray-Smith, Control Theoretic Models of Pointing, ACM Transactions on Computer-Human Interaction, Vol. 24, No. 4, August 2017. http://www.dcs.gla.ac.uk/rod/publications/MueOulMur17.pdf

Murray-Smith, Control Theory, Dynamics and Continuous Interaction, in Computational Interaction Design, eds. A. Oulasvirta, P. O. Kristensson, A. Howes, X. Bi, Oxford University Press, 2018. http://www.dcs.gla.ac.uk/_{rod/publications/Mur18.pdf}

Boland, D., R. McLachlan, R. Murray-Smith. Engaging with mobile music retrieval, Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM , 2015. The B&O Beomoment system which resulted from this work.

B. Vad, Boland, D., Williamson, J., Murray-Smith, R., and Steffensen, P. B., Design and evaluation of a probabilistic music projection interface, In: 16th International Society for Music Information Retrieval Conference, Malaga, Spain, 26-30 Oct 2015. pdf

F Tonolini, BS Jensen, R Murray-Smith, Variational Sparse Coding, Uncertainty in Artificial Intelligence, 2019

Optics & ML: P Caramazza, O Moran, R Murray-Smith, D Faccio, Transmission of natural scene images through a multimode fibre, Nature communications 10 (1), 1-6, 2019.

O. Moran, P. Caramazza, D. Faccio, R. Murray-Smith, Deep, complex, invertible networks for inversion of transmission effects in multimode optical fibres, Advances in Neural Information Processing Systems 31 (NeurIPS 2018), Montreal. pdf

N. Radwell, S. D. Johnson, M. P. Edgar, C. F. Higham, R. Murray-Smith, and M. J. Padgett Deep learning optimized single-pixel LiDAR, Applied Physics Letters 115, 231101 (2019). pdf

C.F. Higham, R. Murray-Smith, M.J. Padgett, M.P. Edgar, Deep learning for real-time single-pixel video, Nature Scientific Reports, 2018. pdf

Nuclear physics & Gaussian processes: D. G. Ireland, M. Döring, D. I. Glazier, J. Haidenbauer, M. Mai, R. Murray-Smith, and D. Rönchen, Kaon Photoproduction and the Λ Decay Parameter α−, Phys. Rev. Lett. 123, 182301, October 2019. pdf

Gravitational waves: H Gabbard, C Messenger, IS Heng, F Tonolini, R Murray-Smith, Bayesian parameter estimation using conditional variational autoencoders for gravitational-wave astronomy, Sept 2019 https://arxiv.org/abs/1909.06296

BIOGRAPHY Roderick Murray-Smith is a Professor of Computing Science at Glasgow University, leading the Inference, Dynamics and Interaction research group, and heads the 60-strong Section on Information, Data and Analysis, which also includes the Information Retrieval, Computer Vision & Autonomous systems and KDE Big Data groups. He works in the overlap between machine learning, interaction design and control theory. In recent years his research has included multimodal sensor-based interaction with mobile devices, mobile spatial interaction, AR/VR, Brain-Computer interaction and nonparametric machine learning. Prior to this he held positions at the Hamilton Institute, NUIM , Technical University of Denmark, M.I.T. (Mike Jordan’s lab), and Daimler-Benz Research, Berlin, and is the Director of SICSA , the Scottish Informatics and Computing Science Alliance (all academic CS departments in Scotland). He works closely with the mobile phone industry, having worked together with Nokia, Samsung, FT/Orange, Microsoft and Bang & Olufsen. He was a member of Nokia’s Scientific Advisory Board and a member of the Scientific Advisory Board for the Finnish Centre of Excellence in Computational Inference Research. He has co-authored three edited volumes, 38 journal papers, 21 book chapters, and over 100 conference papers.

http://www.dcs.gla.ac.uk/rod/ http://www.dcs.gla.ac.uk/~rod/Publications.htm https://twitter.com/IDAglasgow https://twitter.com/MurraySmithRod

This talk is part of the ML@CL Ad-hoc Seminar Series series.

This talk is included in these lists:

• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge talks
• Chris Davis' list
• Computer Laboratory, William Gates Building, Room FW26
• Guy Emerson's list
• Interested Talks
• Interested Talks
• Trust & Technology Initiative - interesting events
• bld31
• ndk22's list
• ob366-ai4er
• rp587

Note that ex-directory lists are not shown.