1 / 23

Gesture recognition using salience detection and concatenated HMMs

Gesture recognition using salience detection and concatenated HMMs. Ying Yin yingyin@csail.mit.edu Randall Davis davis@csail.mit.edu Massachusetts Institute of Technology. System overview. Feature vector sequence. Depth & RGB images. Hand tracking. Hand movement segmentation. Xsens

shiloh
Download Presentation

Gesture recognition using salience detection and concatenated HMMs

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Gesture recognition using salience detection and concatenated HMMs Ying Yin yingyin@csail.mit.edu Randall Davis davis@csail.mit.edu Massachusetts Institute of Technology

  2. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

  3. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

  4. Hand tracking • Kinect skeleton tracking is less accurate when the hands are close to the body or move fast • We use both RGB and depth information • Skin • Gesture salience (motion and closeness to the observer)

  5. Hand tracking

  6. Hand tracking

  7. Input to recognizer • Feature vector xt • From the Kinect data and hand tracking • Relative position of the gesturing hand with respect to shoulder center in world coordinate (R3) • From the Xsens unit on the hand • Linear acceleration (R3) • Angular velocity (R3) • Euler orientation (yaw, pitch, roll) (R3)

  8. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

  9. Hand movement segmentation • Part of gesture spotting • Train Gaussian models for rest and non-rest positions • During recognition, an observation xt is first classified as a rest or a non-rest position • It is a non-rest position if

  10. System overview Feature vector sequence Depth & RGB images Hand tracking Hand movement segmentation Xsens data Feature vector sequence with movement Gesture spotting & recognition

  11. Temporal model of gestures

  12. Continuous gesture models Pre- stroke Post-stroke Nucleus Rest End

  13. Continuous gesture models Pre- stroke Post-stroke Nucleus Rest End

  14. Continuous gesture models Pre- stroke Post-stroke Nucleus Rest End

  15. Pre-stroke & post-stroke phases

  16. Bakis model for nucleus phase • 6 hidden states per nucleus phase in the final model • Emission probability: mixture of Gaussians with 6 mixtures s1 s2 s3 s4 s5 s6 start p(END|s6) p(s1)

  17. Concatenated HMMs • Train an HMM for each phase for each gesture • Model termination probability for each hidden state sas p(END|s) • EM parameter estimation

  18. Concatenated HMMs • After training, concatenate HMMs for each phase to form one HMM for each gesture • Compute transition probability from the previous phase to the next phase • Ensure

  19. Gesture spotting & recognition no nucleus phase • Detect rest vs non-rest segments • Find concatenated HMM that gives the highest probability • Find most probable hidden state sequence using Viterbi • Assign hidden states to corresponding phases • Identify segment without nucleus phase

  20. Gesture recognition result visualization

  21. Gesture recognition result • 10 users and 10 gestures and 3 rest positions • 3-fold average

  22. Gesture recognition result • User independent training and testing • 3-fold average

  23. Contributions • Employed novel gesture phase differentiation using concatenated HMMs • Used hidden states to • identify movements with no nucleus phases • accurately detect start and end of nucleus phases • Improved hand tracking when the hand is close to the body or moving fast by gesture salience detection

More Related