1 / 35

A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor

A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor. Xin Zhang, Zhichao Ye, Lianwen Jin, Ziyong Feng, and Shaojie Xu. FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, Shaojie Xu

mickey
Download Presentation

A New Writing Experience : Finger Writing in the Air Using a Kinect Sensor

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A New Writing Experience :Finger Writing in the Air Using a Kinect Sensor Xin Zhang, ZhichaoYe, LianwenJin, Ziyong Feng, and Shaojie Xu FINGER-WRITING-IN-THE-AIR SYSTEM USING KINECT SENSOR Zhichao Ye, Xin Zhang, Lianwen Jin, Ziyong Feng, ShaojieXu IEEE International Conference on Multimedia and Expo Workshops (ICMEW), 2013 MultiMedia, IEEE, 2013

  2. Outline • Introduction • Related Work • Proposed Method • Experimental Results • Conclusion

  3. Introduction

  4. Introduction • So far most of writing systems still rely on: • Keyboard • Touch screen • …(Extra devices) • Essential goal of HCI: • Making interaction between user and computer more natural

  5. Introduction • In this paper: • Propose a finger-writing-in-the-air system (based on Kinect): • Using depth, color and motion information • Real-time • User-friendly and unconstrained

  6. Related Work

  7. Related work • Hand Segmentation • Skin color: • Gaussian (mixture) model[2] • Illumination and hand-face overlapping • Depth: • noise • Motion: • Motion Cue[3] • The hand should be the most distinct moving object. X X X

  8. Related work [1] L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision based finger-writing character recognition system. Journal of JCSC, 16(3):421–436, 2007. [2] S. L. Phung, A. Bouzerdoum, and D. Chai. Skin segmentation using color pixel classification: Analysis and comparison. IEEE Trans. on PAMI, 27:148–154, 2005. [3] Jonathan Alon, VassilisAthitsos, Quan Yuan and Stan Sclaroff. A Unified Framework for Gesture Recognition and Spatiotemporal Gesture Segmentation. IEEE Trans. on PAMI, 31:1685–1699, 2009. [6] D. Lee and S. Lee. Vision-based finger action recognition by angle detection and contour analysis. Journal of ETRI, 33(3):415–422, 2011. • Fingertip Detection • Curvature[6] • Template matching[1] • Geodesic distance

  9. [10] Ziyong Feng, Shaojie Xu, Xin Zhang, Lianwen Jin, Zhichao Ye and WeixinYang. Real-time Fingertip Tracking and Detection using Kinect Depth Sensor for a New Writing-in the Air System. In Proc. of IEEE ICIMCS, 2012. Related work • Writing-in-the-air system [10]: K-means Arm point • Hand Segmentation • Data Conversion • Region Clustering • Fingertip Identification Fingertip

  10. ProposedMethod

  11. Flow Chart • Hand Segmentation Fingertip Detection

  12. Hand Segmentation • DSB-MM segmentation algorithm

  13. Hand Segmentation depth • Depth Model • Solve the issues: • lighting • hand-face overlapping • moving background • Hand D: R(n) : hand region at frame n ω : : growth factor ↑ ↑

  14. Hand Segmentation • Depth Model A moving hand A static hand

  15. Hand Segmentation • Skin Model • YCbCr color space • Quantify Y Component into three regions: • Bright • Normal • Dark • Gaussian classifier[2]: Reduce the storage size : mean vector of the i-thskin class covariance of the i-thskin class mean vector of the i-thnon-skin covariance of the i-thnon-skin class skin Non-skin (Squared Mahalanobisdistance)

  16. Hand Segmentation • Skin Model Color Image Depth Model Skin Model Depth + Skin

  17. Hand Segmentation [8] K. Kim, T. H. Chalidabhongse, D. Harwood, and L. Davis. Real time foreground-background segmentation using code book model. Real-Time Imaging, 11:172–185, 2005. • Background Model • Codebook background model[8]

  18. Hand Segmentation • Background Model • Codebook background model[8] Color image A Color image B Foreground resultA Foreground resultB

  19. Hand Segmentation • DSB-MM segmentation algorithm

  20. Hand Segmentation • DSB-MM segmentation algorithm • Each model should have different reliabilities. • Adaptive voting system • A pixel is kept as hand pixel by

  21. Hand Segmentation • Artificial Neural Network (ANN) • (1) All the models contribute to the final result. • (2) None of them is absolutely reliable. “1 0 0”, “0 1 0” or “0 0 1” representing 1/3, 1/2 or 2/3 Training: resilient back propagation algorithm (RPROP)

  22. Hand Segmentation Origin Depth Skin Background Mixture

  23. Flow Chart • Hand Segmentation Fingertip Detection

  24. Fingertip Detection • Side-mode & Frontal-mode --(Red) : Side-mode ㄧ(Blue) : Frontal-mode

  25. Fingertip Detection • Side-mode • Fingertip : the farthest point from the arm point • Palm point: • Ellipse fitting technique (center point) • Arm point: • The center of the increased region

  26. Fingertip Detection • Side-mode • The farthest distance to the arm point: • Side-Mode Criterion:

  27. Fingertip Detection • Frontal-mode • Fingertip : the point with the smallest depth value

  28. ExperimentalResults

  29. Experimental Results • Intel Core i5-2400 CPU • 3.10 GHz and 4 Gbytes of RAM • 20 frames per second(fps) • 375 videos(44522 frames) • Recognition of the classifier: • 6763 frequently used Chinese character • 26 English letters (upper case & lower case) • 10 digits

  30. Experimental Results • Finger-writing character recognition • Linking all detected fingertip positions + mean filter • Modified quadratic discriminant function (MQDF) character classifier[9] [9] T. Long and L. Jin. Building Compact MQDF Classifier for Large Character Set Recognition by Subspace Distribution Sharing. Pattern Recognition, 41(9):2916-2926, 2008.

  31. Experimental Results • Error distance (Fingertip detection):

  32. Experimental Results

  33. Experimental Results

  34. Conclusion

  35. Conclusion • Propose a real-time finger-writing-in-the-air system • Hand Segmentation: • Depth + Skin + Motion • Adaptive depth threshold of hand region • Fingertip Detection: • Side-mode • Frontal-mode   

More Related