Sketch Recognition with Multiscale Stochastic Models of Temporal Patterns

Add to your list(s) Download to your calendar using vCal

• Dr Metin Sezgin, Computer Laboratory, University of Cambridge
• Thursday 26 October 2006, 14:15-15:00
• Rainbow Room, Computer Laboratory.

If you have a question about this talk, please contact Daniel Bernhardt.

Sketching is a natural mode of interaction used in a variety of settings. For example, people sketch during early design and brainstorming sessions to guide the thought process; when we communicate certain ideas, we use sketching as an additional modality to convey ideas that can not be put in words. The emergence of hardware such as PDAs and Tablet PCs has enabled capturing freehand sketches, enabling the routine use of sketching as an additional human-computer interaction modality.

But despite the availability of pen based information capture hardware, relatively little effort has been put into developing software capable of understanding and reasoning about sketches. To date, most approaches to sketch recognition have treated sketches as images (i.e., static finished products) and have applied vision algorithms for recognition. However, unlike images, sketches are produced incrementally and interactively, one stroke at a time and their processing should take advantage of this.

In this talk, I will describe ways of doing sketch recognition by extracting as much information as possible from temporal patterns that appear during sketching. I will present a sketch recognition framework based on hierarchical statistical models of temporal patterns. I will show that in certain domains, stroke orderings used in the course of drawing individual objects contain temporal patterns that can aid recognition. Build on this work, I illustrate how sketch recognition systems can use knowledge of both common stroke orderings and common object orderings. I will present a statistical framework based on Dynamic Bayesian Networks that can learn temporal models of object-level and stroke-level patterns for recognition. This framework supports multiobject strokes, multi-stroke objects, and allows interspersed drawing of objects – relaxing the assumption that objects are drawn one at a time. The system also supports real-valued feature representations using a numerically stable recognition algorithm. I will present recognition results for hand-drawn electronic circuit diagrams. The results show that modeling temporal patterns at multiple scales provides a significant increase in correct recognition rates, with no added computational penalties.

This talk is part of the Rainbow Interaction Seminars series.

This talk is included in these lists:

• Cambridge talks
• Human-Computer Interaction
• Rainbow Interaction Seminars
• Rainbow Room, Computer Laboratory
• Trust & Technology Initiative - interesting events

Note that ex-directory lists are not shown.