A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction

1. A Fast and Robust Fingertips Tracking Algorithm for Vision-Based Multi-touch Interaction Qunqun Xie, Guoyuan Liang, Cheng Tang, and Xinyu Wu 2013 10th IEEE International Conference on Control and Automation (ICCA)

2. Outline • Introduction • Related Work • Proposed Method • Hand localization • Fingertips tracking • The Multi-touch system • Experimental Results • Conclusion

3. Introduction

4. Introduction • Multi-touch technology: • Sensor Based • Directly receive finger touch as input • High cost → limits its application to some extent • Computer Vision Based • Good scalability as well as good performance Image: Oka, K, Sato, Y, Koike, H. "Real-time fingertip tracking and gesture recognition", IEEE Computer Graphics and Applications, 2012

5. Introduction • In this paper: • Propose a robust fingertip tracking algorithm: • Real-time • Stereovision-based 3D multi-touch interaction system • Skin / Depth / Geometry structure

6. Related Work

7. Related work L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, 2007. • Geometry properties: • Curvature • Edge or shape • Build a model • Image Analysis • Template matching • Color Segmentation D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.

8. Related work • Palm Center: • Fingertip Detection [b] [a] Geodesic distance GSP points Neighbor depth [a] [d] [c]

9. Related work • [a] Hui Liang, Junsong Yuan, and Daniel Thalmann, "3D Fingertip and Palm Tracking in Depth Image Sequences", Proceedings of the 20th ACM international conference on Multimedia, 2012 • [b]Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-Pao Tsai, and Shawmin Lei, "Real-time Hand Tracking on Depth Images", IEEE Visual Communications and Image Processing (VCIP), 2011 • [c] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye, and Weixin Yang, “Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System”, Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012 • [d] Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu, "FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR", IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013

10. ProposedMethod

11. Hand Segmentation Training data: • Skin Color filter • YCbCr color space • Gaussian Mixture Model • Describe the skin-color distribute • Single Gaussian Model: • Gaussian Mixture Model: Weight of each Gaussian model: color vector

12. Hand Segmentation • Skin Color filter • : how skin-like the color is • Expectation Maximization(EM) algorithm

13. Hand Segmentation • Depth Cue: • The points with minimum depth are picked as seeds • Region grow algorithm skin depth skin + depth

14. Hand Segmentation • Divide wrist and hand: • By a boundary curve [18] • Minimum depth • Boundary curve r: row index c :column index z(r,c) :depth value , range threshold (related to palm size) boundary [18] Z. Mo and U. Neumann, “Real-time hand pose recognition using low-resolution depth images,” in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 2.

15. Palm Region Extraction • Observation : Palm is a rectangle-like region • Method :Project the hand region in all directions

16. Palm Region Extraction Intersection

17. Palm Center Localization X > 、、 > >

18. Palm Center Localization • Palm Center: • The point with maximum distance from the closest palm boundary[18]. • The size of palm R: palm region palm boundary

19. Fingertip Localization • Fingertip : The point with maximum distance to the palm center (on the contour of each finger) • Candidate set F: P : contour point C0: palm center d2:distance R:palm size 1.2 F

20. Fingertip Localization • Assign an index to each point in candidate set: • Sort candidate set by • : index • F : candidate set • C0: palm center • the angle of with negative x-axis

21. Fingertip Localization • Distance between successive points : • If > → Start & End point subset • Fingertips : maximum distance in each subset

22. Multi-touch system • TUIO (Tangible User Interface Object) [24] M. Kaltenbrunner, T. Bovermann, R. Bencina, and E. Costanza, “Tuio:A protocol for table-top tangible user interfaces,” in Proc. of the The 6th Intl Workshop on Gesture in Human-Computer Interaction and Simulation, 2005.

23. ExperimentalResults

24. Experimental Results • Xeon 3.07Ghz workstation • frame rate :20Hz on average(real-time) • Modules • Fingertip tracking • TUIO server • TUIO client [10] C. Shan, Y. Wei, T. Tan, F. Ojardias, ”Real Time Hand Tracking by Combining Particle Filtering and Mean Shift”, In: International Conference on Automatic Face and Gesture Recognition, 2004, pp. 669-674

27. Conclusion

28. Conclusion • Fast and robust fingertip tracking • Without pressuring sensing device & extra marks • Hand Segmentation • Depth / Skin • Fingertip Detection • Palm region projection • Palm center distance from the boundary • Fingertip : assign index (angle)