Real-time continuous gesture recognition for natural human-computer interaction

Abstract

Our real-time continuous gesture recognition system addresses problems that have previously been neglected: handling both gestures that are characterized by distinct paths and gestures characterized by distinct hand poses; and determining how and when the system should respond to gestures. Our probabilistic recognition framework based on hidden Markov models (HMMs) unifies the recognition of the two forms of gestures. Using information from the hidden states in the HMM, we can identify different gesture phases: the pre-stroke, the nucleus and the post-stroke phases. This allows the system to respond appropriately to both gestures that require a discrete response and those needing a continuous response. Our system is extensible: in only a few minutes, users can define their own gestures by giving a few examples rather than writing code. We also collected a new gesture dataset that contains the two forms of gestures, and propose a new hybrid performance metric for evaluating gesture recognition methods for real-time interaction.