1 / 41

Vision-Based Finger Detection and Its Applications 基於電腦視覺之手指偵測及其應用

Vision-Based Finger Detection and Its Applications 基於電腦視覺之手指偵測及其應用. Yi-Fan Chuang Advisor: Prof. Yi-Ping Hung Prof. Ming-Sui Lee. Outline. Introduction Related Work Fingertip Detection and Tracking Applications i-m- Top Magic Crystal Ball (MaC Ball)

redell
Download Presentation

Vision-Based Finger Detection and Its Applications 基於電腦視覺之手指偵測及其應用

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Vision-Based Finger Detectionand Its Applications基於電腦視覺之手指偵測及其應用 Yi-Fan Chuang Advisor: Prof. Yi-Ping Hung Prof. Ming-Sui Lee

  2. Outline • Introduction • Related Work • Fingertip Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  3. Outline • Introduction • Related Work • Fingertip Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  4. Introduction • Interactive system • Bare-handed operations are more intuitive to manipulate digital objects directly • Interactions • Users: hover/touch gestures • Objects: rotation, scaling, selection, special effects…. • Use computer vision techniques • Fingertip detection • Fingertip tracking

  5. Outline • Introduction • Related Work • Fingertip Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  6. Related Work • Enhance Desk • Fingertip finding • Template matching • Palm finding • Morphological erosion H. KOIKE and Y. KOBAYASHI, “Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system,“ ACM Transation Computer-Human Interaction, vol. 8, no. 4, pp. 307-322, 2001.

  7. Related Work • FTIR (Frustrated Total Internal Reflection) J. Y. Han, “Low-cost multi-touch sensing through frustrated total internal reflection," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 115-118.

  8. Related Work • Visual Touchpad • Fingertip detection • Curvature of contour • Finger Orientation • Touch detection • Two warp images S. Malik and J. Laszlo, “Visual touchpad: a two-handed gestural input device," in Proceedings of the 6th international conference on Multimodal interfaces (ICMI '04). New York, NY, USA: ACM Press, 2004, pp. 289-296.

  9. Related Work • PlayAnyWhere • Touch and Hover • Use appearance of shadows A. D. Wilson, “Playanywhere: a compact interactive tabletop projection-vision system," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 83-92.

  10. Related Work • Comparison Our Work: Touch/Hover (1 camera) Multi-user with multi-finger

  11. Outline • Introduction • Related Work • Fingertip Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  12. Overview • System setup • An IR camera with IR illuminators to observe hands • Direct observation • Clear shapes • Indirect observation • Unclear shapes J. Rekimoto and N. Matsushita, “Perceptual surfaces: Towards a human and object sensitive interactive display," Workshop on Perceptural User Interfaces (PUI'97), 1997.

  13. Overview Clear shapes (MaC Ball) • System setup • Direct observation • Clear shapes • Indirect observation • Unclear shapes Captured image Gesture

  14. Overview Unclear shapes (i-m-Top) Captured image Gesture • System setup • Direct observation • Clear shapes • Indirect observation • Unclear shapes

  15. Overview • Fingertip detection • Fingertip tracking Morphological opening Difference of previous two Backgroundsubtraction Principalcomponentanalysis Fingertipdetection Kalman filtering Detection results

  16. Fingertip Detection • Background subtraction • Observation • Contact area leaves strong reflection Backgroundsubtraction Morphological opening Difference of previous two Fingertipdetection PCA (Enhanced image)

  17. Fingertip Detection • Extract finger part • Use a morphological opening operation • The size of structuring element is larger than that of a normal finger and smaller than a palm. Backgroundsubtraction Morphological opening Difference of previous two Fingertipdetection PCA Finger Original Morphological opening (Enhanced image)

  18. Fingertip Detection • Difference and binarization • Principal component analysis (Enhanced image) Backgroundsubtraction Backgroundsubtraction Morphological opening Morphological opening Difference of previous two Difference of previous two Fingertipdetection Fingertipdetection PCA PCA Finger direction Possible fingertips’ positions

  19. Fingertip Detection • Template matching (pattern matching) • Template • Remove false matchings • Distance check • Diagonal check Backgroundsubtraction Morphological opening Difference of previous two Fingertipdetection PCA Finger patch

  20. Background subtraction Morphologicalopening Difference of previous two & PCA Fingertipdetection Detection Results on i-m-Top Separate finger touching and palm hovering (Enhanced image)

  21. Detection Results on MaC Ball Background subtraction Morphological opening Difference of previous two & PCA Fingertipdetection

  22. Fingertip Tracking • Kalman filter • Smooth the path • Predict the new state and its uncertainty • Correct the tracker with its new measurement • Assume white noise and uniform velocity Original After Kalman filter

  23. Performance Evaluation • Real-time system is possible

  24. Outline • Introduction • Related Work • Fingertips Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  25. Hardware Configuration • i-m-Top • Interactive Multi-resolution Tabletop Cooperate with Yi-Wei Chia

  26. Software Implementation • Fingertip detection and tracking • Palm detection and tracking • Association between fingertips and palms • Pen detection • Priority of pens and palms • Events definitions

  27. Palm Detection and Tracking Background subtraction Morphological opening (Palm part) Palmdetection • Palm detection • The average position is picked as palm position • Palm tracking • Kalman filtering (Enhanced Image)

  28. Association between Fingertips and Palms • Allocate fingertips to palms • Use the angle and the distance between fingers and palms (Enhanced Image)

  29. Pen Detection and Priority • Pen Detection • Use a higher threshold • Priority of pens and palms • Higher priority for the pen tip • If there is a palm around the pen tip, this palm and the fingertips associated to the palm will be ignored.

  30. Events Definitions • Hover • Palm position • Touch • Palm position • Finger position and its correspondent palm • Pen position

  31. Application I: Browsing and Editing • Operations • Drag photo/document • Zoom in and zoom out • Stroke • Move the foveal area

  32. Top button Application II:Video Retrieval System • Operations • Query • Browse • Feedback Video plane Video wall Scroll bar Video

  33. Outline • Introduction • Related Work • Fingertip Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  34. Magic Crystal Ball (MaC Ball) • An interactive 3D display • Slide fingers (like a wizard) on the ball surface to control the content

  35. Hardware Configuration User Transparent Glass Ball Fresnel Lens Pressure Sensors Reflection Mirror IR Camera IR Illuminator LCD Detection Module Display Module

  36. Software Implementation • Fingertip detection and tracking • Detect fingertips’ positions • Pointing gesture • Motion detection • Optical flow algorithm • Waving gesture • Contact Detection • Pressure Sensor (Flexi Force) • Detect whether users touch MaC Ball

  37. Software Implementation • Contact detection – Pressure Sensors • Observation p2 p2 p2 p1 p1 p1 p3

  38. Events Definitions • Gesture switch FingertipsDetection MotionDetection FingertipsDetection MotionDetection Pressuresensors Touch / Hover Pointing Gesture Waving Gesture

  39. Application:Virtual Exhibition • Operations • Rotate • Select • Magnifier • Change relics Video

  40. Outline • Introduction • Related Work • Fingertip Detection and Tracking • Applications • i-m-Top • Magic Crystal Ball (MaC Ball) • Conclusion & Future Work

  41. Conclusion & Future Work • Multi-finger detection techniques • Simple and real-time • Cost-effective and flexible • Applications • i-m-Top: interactive interface • MaC Ball: interactive 3D display • Future work • Use FTIR to enhance the accuracy of fingertip detection • More gestures for interactive systems

More Related