Real Time Gesture Learning and Recognition: Towards Automatic Categorization

1. Real Time Gesture Learning and Recognition: Towards Automatic Categorization Jean BaptisteThiebaut, SamerAbdallah, Andrew Robertson, Nick Bryan Kinns, Mark Plumbley Proc. of the 8th Int. Conf. on New Interfaces for Musical Expression(NIME '08) , pp. 215-217, Genova, Italy, June 2008

2. Introduction • Domain • Gesture as an integral part of music performance • Important issue to address is to categorize and recognize gestures • Related works • Research by Cadozand Wanderley [3] • Stress of the importance of gesture classification and recognition • Research by Cadoz[2] • Emphasis of the importance of hapticfeedback for the design of interactive interface for sound production: the physical feedback given by the intermediary device • Creation of memorizablegestures, and audio feedback rendered by the interface • Kelaet al. [5] • Using accelerometers for multi modal activities • Contribution • Real-time classification for use in music performances • Other approaches → pre-defined classification

3. Introduction • Contribution • Development of learning of specific gestures • Creation of a database of recognizable gestures that shared between performers • Two methods • Recognition of a fixed length gesture • Dynamic and unsupervised recognition model to handle various length gestures

4. Supervised Method With Haptic Feedback • Wii remote controller • Popular and pervasive device that detects 3-dimensional movements • Transmission of signals of the accelerometers via Bluetooth • Using Max/MSP to decode the transmissions from the controller • Developed by Masayuki Akamatsu • Sampling rate of 50 Hz • Latency produced by bluetooth of approximate 50ms • Example of data over a fixed period of time • Vibration as feedback to the user when a gesture is recognized • Categorization of a gesture in real time with supervision

5. Euclidean distance for classification • Fragments of the incoming data • L → length of samples depends on the sampling rate • E.g. 6 (x, y, z) triplets at 50 Hz for a gesture → lasting 120ms • Capture of a gesture • Pressing a button (‘A’) on the controller at the end of the movement • Variables • Vr : reference gesture, 3L-dimensional vector • Vi : similar vector constructed from the last L samples of the input signal • 3 차원 벡터 내적의 합 (L 개의 샘플로부터 계산)

6. Cosine similarity for classification • Using cosine of the angle • Calculation of cosine between reference vector and input vector • Taking the dot product and dividing by the norms of the two vectors • Discussion • Cosine method : detection of all 45 instances • Distance method : detection of 44 among 45 instances

7. Unsupervised method using information dynamics • Requirements of above supervised method • Reference gesture with its label • Indication of the particular time point, relative to the reference • Mark indicating the ‘perceptual center’ of the gesture • Predictive information • Heart of gesture recognition → perception of discrete and punctual events in a continuous signal • Predictive information rate of the signal as processed by the observer • Hypothetical observer to be engaged in a continuous process of trying to predict the future evolution of a signal as it unfolds • Assignments of probabilities to the various possible future developments

8. Unsupervised method using information dynamics • Predictive information • Statistical regularities in the signal • E.g.) smoothness or any typical or repeated behavior for better predictions • Kullback-Leiblerdivergence • a measure of distance between probability distributions • Calculation of the divergence between P(Y |Z =z) and P(Y |Z =z,X =x) • where Z =z and X = x denote the propositions that past and present variables respectively were observed to have particular values z and x • Many forms of predictive information rate • In some cases be relatively flat, while in others, more peaky or bursty • Arrival in concentrated ‘packets’ interspersed by longer periods of relatively low predictive information • Identification of the ‘packets’ of information as the ‘events’

9. Unsupervised method using information dynamics • HMMbasedimplementation • Implementation of a version of hypothetical observer • Using a relatively simple predictive model (a Markov chain) • A vector with N = 3L components (L : consecutive samples) • The sequence of vectors → the continuous valued observation sequence • Hidden Markov model (HMM) with Gaussian state-conditional distributions • K possible states • Training of parameters of HMM • Transition matrix and the mean and covariance for each of the K states • Using a variant of the Baum-Welch algorithm • After training, the most likely sequence of hidden states is inferred using the Viterbialgorithm • Variation in predictive information rate over time • Event detection by picking all transitions with a predictive information greater than a fixed threshold

11. Conclusion • Development of efficient tools for real-time gesture recognition • Using Nintendo Wii remote • Both supervised and unsupervised algorithms to deal with signals • Template matching system is based on well-known template matching methods • HMM based system uses novel information-theoretic criteria to enable unsupervised identification of an initially unknown number of gestures • Explicit probabilistic formulation • Handling the detection latency problem by predicting the future motion of the controller • Estimating how accurate this prediction might be

12. Real time gesture recognition using Continuous Time Recurrent Neural Networks Gonzalo Bailador, Daniel Roggen, Gerhard Tröster, a ndGraciánTriviño Proc. of the ICST 2nd int. conf. on Body area networks table of contents, Florence, Italy, 2007

13. INTRODUCTION • Motivation • Appearance of intelligent devices integrating in clothing or objects • “On the body” wearable computers • Ideal location to detect important information about the “state” of the user • Such as his position, his activities or gestures or even his social interactions • Fields for context awareness • Personal health assistant of the user (e.g. by monitoring the physical activity) • Delivering context-based information • Human-computer interactions • Detect social interactions • Insight into affective disorders or depression • Challenge of gesture recognition in wearable computing • Good recognition accuracy on miniature wearable devices • Offering long battery life, and consequently limited computational power

14. INTRODUCTION • Technologies for gesture recognition • Hidden Markov models (predominant approach) • Dynamic programming • Neural networks • Fuzzy expert systems • Analyzing complex features of the signal like the dopplerspectrum • Difficult to classify two different gestures with similar values for these features