A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor

1. A New Writing Experience :Finger Writing in the Air Using a Kinect Sensor Xin Zhang, ZhichaoYe, LianwenJin, Ziyong Feng, and Shaojie Xu FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, ShaojieXu IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013 MultiMedia, IEEE, 2013

2. Outline • Introduction • Related Work • Proposed Method • Experimental Results • Conclusion

3. Introduction

4. Introduction • So far most of writing systems still rely on: • Keyboard • Touch screen • …(Extra devices) • Essential goal of HCI: • Making interaction between user and computer more natural

5. Introduction • In this paper: • Propose a finger-writing-in-the-air system (based on Kinect): • Using depth, color and motion information • Real-time • User-friendly and unconstrained

6. Related Work

7. Related work • Hand Segmentation • Skin color: • Gaussian (mixture) model[2] • Illumination and hand-face overlapping • Depth: • noise • Motion: • Motion Cue[3] • The hand should be the most distinct moving object. X X X

8. Related work [1] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of JCSC, 16(3):421–436, 2007. [2] S. L. Phung, A. Bouzerdoum, and D. Chai. Skin segmentation using color pixel classification: Analysis and comparison. IEEE Trans. on PAMI, 27:148–154, 2005. [3] Jonathan Alon, VassilisAthitsos, Quan Yuan and Stan Sclaroff. A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation. IEEE Trans. on PAMI, 31:1685–1699, 2009. [6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Journal of ETRI, 33(3):415–422, 2011. • Fingertip Detection • Curvature[6] • Template matching[1] • Geodesic distance

9. [10] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye and WeixinYang. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in the Air System. In Proc. of IEEE ICIMCS, 2012. Related work • Writing-in-the-air system [10]: K-means Arm point • Hand Segmentation • Data Conversion • Region Clustering • Fingertip Identification Fingertip

10. ProposedMethod

11. Flow Chart • Hand Segmentation Fingertip Detection

12. Hand Segmentation • DSB-MM segmentation algorithm

13. Hand Segmentation depth • Depth Model • Solve the issues: • lighting • hand-face overlapping • moving background • Hand D: R(n) : hand region at frame n ω : : growth factor ↑ ↑

14. Hand Segmentation • Depth Model A moving hand A static hand

15. Hand Segmentation • Skin Model • YCbCr color space • Quantify Y Component into three regions: • Bright • Normal • Dark • Gaussian classifier[2]: Reduce the storage size : mean vector of the i-thskin class covariance of the i-thskin class mean vector of the i-thnon-skin covariance of the i-thnon-skin class skin Non-skin (Squared Mahalanobisdistance)

16. Hand Segmentation • Skin Model Color Image Depth Model Skin Model Depth + Skin

17. Hand Segmentation [8] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005. • Background Model • Codebook background model[8]

18. Hand Segmentation • Background Model • Codebook background model[8] Color image A Color image B Foreground resultA Foreground resultB

19. Hand Segmentation • DSB-MM segmentation algorithm

20. Hand Segmentation • DSB-MM segmentation algorithm • Each model should have different reliabilities. • Adaptive voting system • A pixel is kept as hand pixel by

21. Hand Segmentation • Artificial Neural Network (ANN) • (1) All the models contribute to the final result. • (2) None of them is absolutely reliable. “1 0 0”, “0 1 0” or “0 0 1” representing 1/3, 1/2 or 2/3 Training: resilient back propagation algorithm (RPROP)

22. Hand Segmentation Origin Depth Skin Background Mixture

23. Flow Chart • Hand Segmentation Fingertip Detection

24. Fingertip Detection • Side-mode & Frontal-mode --(Red) : Side-mode ㄧ(Blue) : Frontal-mode

25. Fingertip Detection • Side-mode • Fingertip : the farthest point from the arm point • Palm point: • Ellipse fitting technique (center point) • Arm point: • The center of the increased region

26. Fingertip Detection • Side-mode • The farthest distance to the arm point: • Side-Mode Criterion:

27. Fingertip Detection • Frontal-mode • Fingertip : the point with the smallest depth value

28. ExperimentalResults

29. Experimental Results • Intel Core i5-2400 CPU • 3.10 GHz and 4 Gbytes of RAM • 20 frames per second(fps) • 375 videos(44522 frames) • Recognition of the classifier: • 6763 frequently used Chinese character • 26 English letters (upper case & lower case) • 10 digits

30. Experimental Results • Finger-writing character recognition • Linking all detected fingertip positions + mean filter • Modified quadratic discriminant function (MQDF) character classifier[9] [9] T. Long and L. Jin. Building Compact MQDF Classifier for Large Character Set Recognition by Subspace Distribution Sharing. Pattern Recognition, 41(9):2916-2926, 2008.

31. Experimental Results • Error distance (Fingertip detection):

32. Experimental Results

33. Experimental Results

34. Conclusion

35. Conclusion • Propose a real-time finger-writing-in-the-air system • Hand Segmentation: • Depth + Skin + Motion • Adaptive depth threshold of hand region • Fingertip Detection: • Side-mode • Frontal-mode   