Ziyong Feng, Shaojie Xu, Xin Zhang , Lianwen Jin, Zhichao Ye, and Weixin Yang

1. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in-the Air System ZiyongFeng, Shaojie Xu, Xin Zhang,Lianwen Jin, ZhichaoYe, and WeixinYang Proceedings of the 4th International Conference on Internet Multimedia Computing and Service, 2012

2. Outline • Introduction • Related Work • Proposed Method • Experimental Results • Conclusion

3. Introduction

4. Introduction • Fingertip detectiontakes a very important role of the natural HCI • Challenge : • Variety of hand poses • Occlusion • In this paper: • Propose a real-time finger writing character recognition system using depth information • Accurate and fast (Human Computer Interaction)

5. Related Work

6. Related work • Template matching[3]: • Curvature Fitting[6]: [3] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of Circuits, Systems, and Computers (JCSC), 16(3):421–436, 2007. [6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Electronics and Telecommunications Research Institute Journal, 33(3):415–422, 2011.

7. ProposedMethod

8. Flow Chart • Hand Segmentation • Data Conversion • Region Clustering • Fingertip Identification

9. Hand Segmentation • Extract human body from background: • User ID map ( by Open Natural Interaction (OpenNI) ) • User Generator

10. Hand Segmentation • Two kinds hand-torso relationship: • 1) Hand is holding up front. • 2) Hand is close to the body. Depth Histogram

11. Hand Segmentation • Characterize the depth-histogram by two models: • 1) Two component Gaussian mixture model . • 2) Single Gaussian model. • Hand pixels : • Belong to the Gaussian component with smaller mean Two-Component •  :weight of k-th component :maenof k-th component :variance of k-th component d :depth value Expectation-maximization algorithm

12. Hand Segmentation • One Gaussian fitting: • When the means of two Gaussian are too near • Distribution: • Hand pixels: • Compared with torso, hand takes a few room. • Lower part of p : One-Component • 

13. Data Conversion • Convert to real world coordinate: • The accuracy of world coordinate is about 1mm. • The following discussions are all based on real-world coordinate. :projected point coordinate d :depth value :camera’s focal length at axis x and y x : real word coordinate

14. Region Clustering • Clustering algorithm : K-means • Finger part vs. non-finger part (K=2) • Minimize distortion measure J: n-th sample would be assigned to k-th cluster maen of the k-th cluster

15. Fingertip Identification • After clustering → hand-related region is separated into two parts. • The fingertip: • The farthest point from one cluster to the center of the other cluster ‧Arm point: -the mean of points that have the same maximum depth ‧The fingertip: O X

16. ExperimentalResults

17. Experimental Results • Resolution : 480 640 • 30 ftps using OpenNI (KINECT) • Dataset: • 2 subjects • 6 categories • Total 8185 frames

18. Experimental Results

19. Experimental Results Near mode (1m) Far mode (1.5m)

20. Experimental Results • The distribution of errors from a sequence: ‧Fast movement ‧Finger is orthogonal to the camera plane.

21. Experimental Results • Smoothed trajectory: Mean filter • 90% recognition rate on English characters • 80% on Chinese characters

22. Conclusion

23. Conclusion • Proposes a novel real-time fingertip detection and tracking. • Using depth sequences • Accurate and fast on fingertip detection & character recgonition

24. Real-time Hand Tracking on Depth Images Chia-Ping Chen, Yu-Ting Chen, Ping-Han Lee, Yu-PaoTsai, and Shawmin Lei Visual Communications and Image Processing (VCIP), 2011 IEEE

25. Outline • Introduction • Proposed Method • Experimental Results • Conclusion

26. Introduction

27. Introduction • Most previous works tracked the hand position on color images and relied heavily on skin colorinformation. • Vulnerable tolighting variations and skin color • In this paper: • Propose a hand tracking algorithm that uses depth images only • Real-time and accurate • Hand click detection method

28. ProposedMethod

29. Hand Position Detection • Predict the new hand position based on the hand moving velocity: • H : hand moving velocity (estimated from hand positions tracked in previous frames)

30. Hand Region Segmentation • Hand region: • Connected component in the 3D point cloud P (from 2D depth image) • Seed Point: • d(.,.) : Euclidean distance • The nearest point in the point cloud P from the predicted hand position ‧Seed Point ‧Predicted hand position

31. Hand Region Segmentation • Connectivity: • Entire hand region: • Using standard region growing techniques • Hand region grows incrementally and stops when: • 1) Two neighboring points are no longer connected • 2) The geodesic distance to the seed point < 30mm Seed Point 250mm

32. Hand Region Segmentation • A) Rough hand center: • --The point with maximum boundary points in its neighborhood • -- There should be more boundary points around the palm. • B) Refined hand center: (12mm) Mean-Shift (One iteration)

33. Hand Region Segmentation • C) Hand center after Mean-Shift:

34. ExperimentalResults

35. Experimental Results • Resolution : 320 240 • 3GHz Intel Core 2 Duo E8400 • Computational complexity:

36. Experimental Results

37. Experimental Results • Ground truth vs. tracked position (in millimeters)

38. Conclusion

39. Conclusion • Proposes a real-time hand tracking algorithm on depth images. • Using: • Region Growing • Geodesic distance • Mean-shift • Can be further extended to two-hand tracking: