1 / 36

Dynamic and static hand gesture recognition in computer vision

Dynamic and static hand gesture recognition in computer vision. Andrzej Czyżewski, Bożena Kostek, Piotr Odya , Bartosz Kunka , Michał Lech Gdansk University of Technology, Faculty of Electronics, Telecommunications and Informatics Multimedia Systems Dept. Warsaw, 13.08.2014.

lucian
Download Presentation

Dynamic and static hand gesture recognition in computer vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dynamic and statichandgesturerecognitionin computer vision Andrzej Czyżewski, Bożena Kostek, Piotr Odya, Bartosz Kunka, Michał Lech GdanskUniversity of Technology, Faculty of Electronics, Telecommunications and Informatics Multimedia Systems Dept. Warsaw, 13.08.2014

  2. Presentationoutline • Developed gesture recognition system • Background / foreground segmentation • Recognizing dynamic hand gestures • Recognizing static hand gestures • Efficiency • Video presentations

  3. Presentationoutline • Developedgesturerecognition system • Background / foregroundsegmentation • Recognizingdynamichandgestures • Recognizingstatichandgestures • Efficiency • Video presentations

  4. DevelopedGesturerecognition system (1) • Features of the gesture recognition system • Recognizing static (palm shape) and dynamic gestures (motion trajectory) of one or both hands • The same dynamic gesture can have various meanings depending on the static gesture • No datagloves, accelerometers or infrared emitters / sensors are needed

  5. DevelopedGesturerecognition system (2) • System components • PC • Webcam (RGB) • Multimedia projector • Screen for projected image • A user stands between a projection screen and the multimedia projector

  6. DevelopedGesturerecognition system (3) Gesture dictionary

  7. DevelopedGesturerecognition system (4) • System working with the developed applications • Virtual Whiteboard application • alternative solution to electronic whiteboards • Gesture-based sound mixing system • new method of sound mixing immersing an engineer more in the sound

  8. Presentationoutline • Developedgesturerecognition system • Background / foregroundsegmentation • Recognizingdynamichandgestures • Recognizingstatichandgestures • Efficiency • Video presentations

  9. Background / foregroundsegmentation (1) • Most crucial part in RGB vision based systems considering gesture recognition efficacy • influences representation of a hand shape in the image • influences the degree of noise in the image – false positive detections

  10. Background / foregroundsegmentation (2) • Two possible scenarios regarding camera placement • front-faced camera placement • back-faced camera placement (environment employing multimedia projector)

  11. Background / foregroundsegmentation (3) • Front-faced camera placement • Varying background behind a user • User free movements • Influence of lighting changes

  12. Background / foregroundsegmentation (4) • Back-faced camera placement • User not visible directly in the image • Background is relatively stable • Influence of lighting changes • Distortions in the image introduced by: • Camera and projector lens • Impact of lighting on displayed image color

  13. Background / foregroundsegmentation (5) • The simplest background subtraction • Principle • calculating a reference (background) image • subtracting each new frame from the reference image • thresholding the difference • Difference image is noisy and very susceptible to lighting changes • More practical approach • to calculate a time-averaged image

  14. Background / foregroundsegmentation (6) • Background modelling • Considering background changes and adaptation • Typical methods: • Codebook • Including periodical changes in the model • No adaptation • GMM • Adaptation to background changes • Skin color modelling • Relatively independent of background changes • Unreliable when background color is similar to skin color • Influence of lighting on skin color

  15. Background / foregroundsegmentation (7) • Background / foreground segmentation in the developed gesture recognition system (camera – projector configuration) • The principle involves absolute subtracting the original image displayed by the multimedia projector from the processed image captured by the camera Processed camera frame Displayed image Resulting image

  16. Background / foregroundsegmentation (8) a) b) c) d) e) f) a) perspective corrected camera image; b) e) image displayed by the projector; c) difference of a and b after converting to gray scale, thresholding and median filtering; d) perspective corrected and color calibrated camera image; f) difference of d and e after converting to gray scale, thresholding and median filtering;

  17. Background / foregroundsegmentation (9) Perspectivecorrectedimage Colorcalibratedcroppedimage Camera image Imageafterconversion to grayscale Imagedisplayed by theprojector Absolutedifferenceresult Binarythresholdedimage Median filteredimage

  18. Presentationoutline • Developedgesturerecognition system • Background / foregroundsegmentation • Recognizingdynamichandgestures • Recognizingstatichandgestures • Efficiency • Video presentations

  19. Recognizingdynamichandgestures (1) Motion modelling based on 2 succesive motion vectors The singular motion vector is designated on points localizing hand in frames n and n + c (c is a function of frame rate and for 22 FPS equals 3)

  20. Recognizingdynamichandgestures (2) • The velocity and direction of the motion is analysed using fuzzy-rule based system • 8 linguistic variables: • The inference zero-order Sugeno model with singletons denoting gesture classes is suitable for dynamic gesture recognition • 30 fuzzy rules • Exemplary rule: // beginning phase of hand movement in the left direction (for semi-circular motion) for left hand RULE 1 : IF directionLt0 IS north AND directionLt1 IS west AND velocityLt0 ISNOT small ANDvelocityLt1 IS NOT small ANDvelocityRt0 IS vsmallAND velocityRt1 ISvsmall THEN gesture IS g1;

  21. Recognizingdynamichandgestures (3) • The outputs of fuzzy rules are filtered with threshold equal to 0.5; below this value the motion activity is not associated with any of the defined gestures • The output of the system is the maximum of all rule outputs • Triangle membership functions used in the process of fuzzification for all variables

  22. Recognizingdynamichandgestures (4) • Description of fuzzyinference module in FCL (FuzzyControlLanguage) // beginningphase of lefthandmotioninrightdirectionRULE 8 : IF directionLt0 IS North AND directionLt1 IS East AND velocityLt0 IS NOT small AND velocityLt1 IS NOT small AND velocityRt0 IS vsmall AND velocityRt1 IS vsmall THEN gesture IS g2; // middlephase of lefthandmotioninrightdirectionRULE 9 : IF directionLt0 IS East AND directionLt1 IS East AND velocityLt0 IS NOT small AND velocityLt1 IS NOT small AND velocityRt0 IS vsmall AND velocityRt1 IS vsmall THEN gesture IS g2; // endingphase of lefthandmotioninrightdirectionRULE 10 : IF directionLt0 IS East AND directionLt1 IS South AND velocityLt0 IS NOT small AND velocityLt1 IS NOT small AND velocityRt0 IS vsmall AND velocityRt1 IS vsmall THEN gesture IS g2;

  23. Recognizingdynamichandgestures (5) Hand tracking supported by Kalman filters

  24. Recognizingdynamichandgestures (6) Examining Kalman filters applied to trajectory smoothing Visualization of motion trajectories obtained for the system with Kalman filters (darker line) and system without Kalman filters (brighter line)

  25. Presentationoutline • Developedgesturerecognition system • Background / foregroundsegmentation • Recognizingdynamichandgestures • Recognizingstatichandgestures • Efficiency • Video presentations

  26. Recognizingstatichandgestures (1) Representing hand shape using PGH PGH Creating Pairwise Geometrical Histogram: a) calculating distances and angles between segments designated on object contour; b) two dimensional PGH (Bradski, 2008) Hand shape parameterized using PGH method (Pairwise Geometrical Histograms)

  27. Recognizingstatichandgestures (2) • To provide reliable gesture recognition it is essential to chose the optimal classifier • experiments using WEKA application • Random Tree • C4.5 (J48) • Naive Bayes Net • NNge • Random Forest • Artifical Neural Network • Support Vector Machines

  28. Recognizingstatichandgestures (3) The results of classifiers examination tT – average training time, tK – average validation time

  29. Recognizingstatichandgestures (4) • The SVM classifier of a type C-SVC (C-Support Vector Classification) with RBF kernel can be considered optimal • The highest efficacy (SVM:92,82%, ANN: 91,67%) • Lack of generalization effect typical for ANN classifier

  30. Presentationoutline • Developedgesturerecognition system • Background / foregroundsegmentation • Recognizingdynamichandgestures • Recognizingstatichandgestures • Efficiency • Video presentations

  31. Efficiency (1) • Computer parameters: • Intel Core 2 Duo P7350 2.0 GHz • 400 MHz DDR2 RAM, 6:6:6:18 cycle latency • Windows Vista Business 32-bit • Screen resolution: 1024 x 768 px • Processing frames of a size 320 x 240 px

  32. Efficiency (2) Averaged execution times of most time consuming operations over 1000 iterations Obtained average frame rate: ~22 FPS

  33. Presentationoutline • Developedgesturerecognition system • Background / foregroundsegmentation • Recognizingdynamichandgestures • Recognizingstatichandgestures • Efficiency • Video presentations

  34. VirtualWhiteboard

  35. GestureMixer

  36. Thankyou for yourattention.

More Related