1 / 31

3D Fingertip and Palm Tracking in Depth Image Sequences

3D Fingertip and Palm Tracking in Depth Image Sequences. Hui Liang , Junsong Yuan and Daniel Thalmann. Proceedings of the 20th ACM international conference on Multimedia 2012. Outline. Introduction Related Work Proposed Method Experimental Results Conclusion. Introduction.

elysia
Download Presentation

3D Fingertip and Palm Tracking in Depth Image Sequences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3D Fingertip and Palm Tracking in Depth Image Sequences Hui Liang, JunsongYuanand Daniel Thalmann Proceedings of the 20th ACM international conference on Multimedia2012

  2. Outline • Introduction • Related Work • Proposed Method • Experimental Results • Conclusion

  3. Introduction

  4. Introduction • Human hand is an essential body part for human-computer interaction. • The positions of tracked fingertips: gesture estimation • Difficulty in fingertip tracking: Side-by-side Bending Nearby

  5. Introduction • Many previous methods: • Only focus on extracting 2Dfingertips • Cannot track fingertips robustly for a freely moving hand • In this paper: • Present a robust fingertip and palm tracking scheme • With the input of depth images (KINECT) • Track the 3D fingertip positions quite accurately

  6. Related Work

  7. Related work • Focus only on 2D fingertips:[4][5][6][9] • Based on contour analysis of the extracted hand region:[2][4][5][6] • Usually can track the fingertips for only stretched fingers.

  8. Related work • In [6], • Fingertips are tracked for infrared image sequences. • It utilizes a template matching strategy • Fingertip tracking : Kalman filter • In [2], • Stereoscopic vision is adopted • Maximize the distance center of gravity of the hand & the boundary

  9. [9] J. L. Raheja, A. Chaudhary, and K. Singal. Tracking of fingertips and centres of palm using kinect, 2011. Proc. Of the 3rd Int’l Conf. on Computational Intelligence, Modelling and Simulation. Related work • In [9] (Kinect), Minimum depth Depth < Threshold Circular filter

  10. Related work • [2] S. Consei1, S. Bourennane, and L. Martin. Three dimensional fingertip tracking in stereovision, 2005. Proc. of the 7th Int’l Conf. on Advanced Concepts for Intelligent Vision Systems. • [4] K. Hsiao, T. Chen, and S. Chien. Fast fingertip positioning by combining particle filtering with particle random diffusion, 2008. Proc. IEEE Int’l Conf. on Multimedia and Expo. • [5] I. Katz, K. Gabayan, and H. Aghajan. A multi-touch surface using multiple cameras, 2007. Proc. of the 9th Int’l Conf. on Advanced concepts for intelligent vision systems. • [6] K. Oka, Y. Sato, and H. Koike. Real-time tracking of multiple fingertips and gesture recognition for augmented desk interface systems, 2002. Proc. IEEE Int’l Conf. on Automatic Face and Gesture Recognition. • [9] J. L. Raheja, A. Chaudhary, and K. Singal. Tracking of fingertips and centres of palm using kinect, 2011. Proc. Of the 3rd Int’l Conf. on Computational Intelligence, Modelling and Simulation.

  11. ProposedMethod

  12. Overview • Stereo images • (Depth map) • Foreground Segmentation • Palm • Localization • Hand • Segmentation • Fingertip Detection • Fingertip Tracking • Fingertips

  13. Hand and Palm Detection • 1) Assume the hand is the nearest object • 2) Constrain global hand rotation by: • : global rotation angle of the hand

  14. Hand and Palm Detection • Foreground Segmentation  • Palm • Localization • Threshold the depth frame to obtain the foregroundF: • p : a pixel coordinate • z(p) : depth value (of point p ) • z0 : the minimum depth value • zD : depth threshold • Hand • Segmentation 0.2m foreground F

  15. Hand and Palm Detection • Foreground Segmentation • Palm • Localization • The palm region is approximated with a circle: • pp: the palm center (of point p ) • rp : the radius • Assume that hand palm forms a globally largest blob • Cpequals to the largest inscribed circle of the contour of F . • 2D Kalman filter : reduce computational complexity • Hand • Segmentation

  16. Hand and Palm Detection • Foreground Segmentation • Palm • Localization • Separate hand and forearmbyaline: • 1) Tangent to Cp • 2) Perpendicular to the orientation of the forearm • Orientation of the forearm : • The Eigenvector that corresponds to the largest Eigenvalue of the covariance matrix of the contour pixel coordinates of F • Hand • Segmentation Hand region : FV(2D)→ FD(3D)

  17. Fingertip Detection and Tracking • Constraints on possible fingertip locations: • 1) Only in depth discontinuous region ( in contour Fv) • 2) | Depth(one point) – Depth(neighborhoods) | are important. • 3) Utilize the 3D geodesic shortest path (GSP) Fingertip vs. Non-fingertip Nearby Fingertips

  18. Fingertip Detection • Fingertip detection • Fingertip tracking • Goal: detect all five fingertips in the depth image • Based on three depth-based features • Build a graph G : • Vh : contains of all points within FV (hand contour) • Eh : for each pair of vertices(p,q), 1) they are in the 8-neighborhood of each other 2) their 3D distance is within threshold τ Edge weight : 3D Euclidean distance

  19. Fingertip Detection • Fingertip detection • Fingertip tracking • Calculate Geodesic distance dg(p): • From palm center ppfor each vertex Vh • Dijkstra graph search on Gh • GSP point set Ug(p): • The set of vertices on the shortest path from pp to p • Rectangle local feature RL(p): • Describe the neighborhood of a point p in FV • : ratio of 1s 1cm

  20. Fingertip Detection • Fingertip detection • Fingertip tracking • Calculate Geodesic distance dg(p): 0.4 dg(p) fingertips

  21. Fingertip Detection • Fingertip detection • Fingertip tracking • Fingertip labeling: Nmax=6 j : label :estimate the probability that has the label lj number of GSP points kth GSP point i : fingertip

  22. Fingertip Detection • Fingertip detection • Fingertip tracking • Fingertip labeling:

  23. Fingertip Tracking • Fingertip detection • Fingertip tracking • Build a particle filter for each fingertip • (x, ω) denote a particle • x : 2D position in FV • ω : the particle weight • Constrain the positions of each particle to the border point set UB to reduce the search space

  24. Fingertip Tracking • Fingertip detection • Fingertip tracking • Likelihood function : pre now Geodesic distance / dg(p) GSP points Neighbor depth /

  25. Fingertip Tracking • Fingertip detection • Fingertip tracking • Likelihood function :

  26. ExperimentalResults

  27. Experimental Results ‧Error : Euclidean distance ‧Ground truth : phalanx end point • Quantitative results on synthetic sequences:

  28. Experimental Results

  29. Experimental Results • Virtual object grasping:

  30. Conclusion

  31. Conclusion • Using multiple depth-based features for accurate fingertip localization • Adopting a particle filter to track the fingertips over successive frames • Track the 3D positions of fingertips robustly • Great potential for extension to other HCI applications

More Related