550 likes | 644 Views
Toolkits for Supporting Gestures in Applications. Justin Weisz 05-830 UI Software Nov. 16, 2004. What is a gesture?. “A single stroke indicates the operation (move text), the operand (the text to be moved), and additional parameters (the new location of the text).” -- Rubine.
E N D
Toolkits for Supporting Gestures in Applications Justin Weisz 05-830 UI Software Nov. 16, 2004
What is a gesture? “A single stroke indicates the operation (move text), the operand (the text to be moved), and additional parameters (the new location of the text).” -- Rubine
What is a gesture? “A single stroke indicates the operation (move text), the operand (the text to be moved), and additional parameters (the new location of the text).” -- Rubine
Editing existing objects Creating new objects Uses of gestures
Issuing commands Back Reload page Menu > Copy Uses of gestures
Applications of gesturing - Video games PowerGlove in “The Wizard”
Applications of gesturing - Video games Black and White - 2001
Rubine [1991] Gesture Recognizers Automated (in a) Novel Direct Manipulation Architecture
Rubine [1991] For the line gesture: recog = [Seq :[handler mousetool:LineCursor] :[[view createLine] setEndpoint:0 x:<start X> y:<start Y>] ]; manip = [recog setEndpoint:1 x:<current X> y:<current Y>]; done = nil;
Rubine [1991] BUT, how are gestures actually represented and recognized? Assumptions: • Gestures are 2D, single strokes • Start and end of a gesture is clearly defined Representation: set of P sample points position & timestamp, preprocessed to remove jitter
Example features: cos of initial angle length of BB diagonal angle between three pts(?) total angle traversed Rubine [1991] Feature vector extracted from G:
Rubine [1991] BUT... “The aforementioned feature set was empirically determined by the author to work well on a number of different gesture sets” -- Rubine
Rubine [1991] Classification Each gesture class represented by a weight vector To classify gesture G: weight of feature i score bias(?) feature i of gesture G Take the highest score:
Rubine [1991] Training Optimal classifier
Rubine [1991] Rejection > 0.95? ACCEPT Gesture G Pr(G matches i) Classification i REJECT g1 g3 g2 mean(i)
Rubine [1991] Evaluation
Rubine [1991] Evaluation
Rubine [1991] Evaluation
gdt - Long et al. [1999] Newton and Palm users reported: • Gestures are powerful, efficient and convenient • Want more commands to have gestures • Want to define new gestures • Recognition accuracy is not good enough
gdt - Long et al. [1999] Oh Agate, I will make you beautiful!
gdt - Long et al. [1999] Distance matrix
gdt - Long et al. [1999] Classification matrix
gdt - Long et al. [1999] Experiment - Hypotheses • “Participants could use gdt to improve their gesture sets.” • “The tables gdt provided would aid designers.” • “PDA users and non-PDA users would perform differently.”
gdt - Long et al. [1999] Experiment - Procedure (pay no attention to the man behind the curtain...)
gdt - Long et al. [1999] Experiment - Results
“Clustering” d Reverse direction existing gesture classes new class gdt - Long et al. [1999] Experiment - Problems with gdt
“Sloppiness” Gesture overloading Delete gdt - Long et al. [1999] Experiment - Problems with gdt
rect copy gdt - Long et al. [1999] Lessons learned • GDT helpful, but participants averaged a 95.4% recognition rate • Tables too confusing, didn’t help performance (better: “Gesture class A is too similar to gesture class B”) • Should be able to create a test set of gestures and run it against a different gesture class
Break time! Muchas gracias to my officemate for the suggestion. Smiling babies make people happy. BE HAPPY!
Problem: Real problem: it is (still) cumbersome to design a system to perform gesture recognition GT2k - Westeyn et al. [2003]
I’m back! Sensors Microphones Cameras Accelerometers <action> GT2k - Westeyn et al. [2003] GT2k system components Data generator Results interpreter
HMM Transition probs. Symbol output probs. Initial state dist. kth symbol in the alphabet Aside: Hidden Markov Models
Aside: Hidden Markov Models • Evaluation problem • Given HMM and O={o1,...,oT}, compute Pr(O|HMM) • Forward algorithm • Decoding problem • Given O, compute most likely state sequence that produced O • Viterbi algorithm • Learning problem • Given O, compute transition probs. to maximize likelihood of observing O • Forward-Backward algorithm (aka. Baum-Welch)
GT2k - Westeyn et al. [2003] Grammars MoveForward = Advance Slow_Down Halt MoveBackward = Reverse Slow_Down Halt command = Attention <MoveForward | MoveBackward>
GT2k - Westeyn et al. [2003] Converting raw sensor data to feature vectors 1 56 Attention 57 175 Advance 176 235 Slow_Down 236 250 Halt
train overfit! test GT2k - Westeyn et al. [2003] Training Training and validation procedure
only during continuous recognition GT2k - Westeyn et al. [2003] Accuracy A = accuracy N = number of examples S = # substitution errors (misclassification) D = # deletion errors (failed to recognize a gesture) I = # insertion errors (system hallucinates a gesture)
GT2k - Westeyn et al. [2003] Applications - Gesture Panel gesture = up | down | left | right | up-left | up-right | down-left | down-right Result: 99.20% accuracy on 251 examples (2 substitution errors)
GT2k - Westeyn et al. [2003] Applications - Prescott blinkprint = person_1 | person_2 | person_3 Result: 89.6% accuracy on 48 examples (5 substitution errors, not good!)
GT2k - Westeyn et al. [2003] Applications - TeleSign word = my | computer | helps | me | talk sentence = ( calibrate word word word word word exit ) Result: 90.48% accuracy on 72 examples
GT2k - Westeyn et al. [2003] Applications - Workshop Activity Recognition gesture = hammer | file | sand | saw | screw | vise | drill | clap | use_drawer | grind Result: 93.33% accuracy on 10 examples per activity