1 / 25

Recognition, Analysis and Synthesis of Gesture Expressivity

Recognition, Analysis and Synthesis of Gesture Expressivity. George Caridakis IVML-ICCS. Overview. Corpus Image processing module Gesture Recognition Expressivity Analysis Expressivity Synthesis Applications. Overview. Corpus mint-IVML. 7 subjects 7 gesture classes

hera
Download Presentation

Recognition, Analysis and Synthesis of Gesture Expressivity

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Recognition, Analysis and Synthesis of Gesture Expressivity George Caridakis IVML-ICCS

  2. Overview • Corpus • Image processing module • Gesture Recognition • Expressivity Analysis • Expressivity Synthesis • Applications

  3. Overview

  4. Corpus mint-IVML • 7 subjects • 7 gesture classes • 20 gesture variations (3 quadrants) • 20’ minutes – 30000 frames

  5. Corpus EmoTV

  6. Corpus GEMEP (on going…)

  7. Head detection • Detect candidate facial areas • Validate using skin probability • Conclude on number of persons

  8. Hand Detection • Skin probability • Thresholding & Morphology Operations • Distance Transform • Frame difference

  9. Tracking • Scoring system based on: • Skin region size • Distance wrt the previous position • Optical flow alignment • Spatial constraints • Thresholding scores • Periodical re-initialization

  10. Head & Hand Tracking

  11. HMM parameters for gestures • States are head and hands coordinates • XL-XR XH-XR XH-XL YL-YR YH-YR YH-YL • 6 output states • Bakis left-to-right models • Continuous output distribution • 3 Gaussian mixtures • Arbitrary training initial estimation of transition probabilities

  12. Recognition via HMM (Why HMMs?) • Stochastic models fit the nature of the gestures • Fast convergence due to effective training algorithms • Sufficient modeling of the temporal aspect of gestures • Continuous HMMs suitable for gesture-level classification

  13. HMM overview

  14. Recognition via HMM

  15. Results

  16. Expressivity features analysis • Overall activation • Spatial extent • Temporal • Fluidity • Power/Energy • Repetitivity

  17. Overall activation • Considered as the quantity of movement during a conversational turn • Computed as the sum of the motion vectors’ norm

  18. Spatial extent • Modeled by expanding or condensing the entire space in front of the agent that is used for gesturing • Calculated as the maximum Euclidean distance of the position of the two hands • The average spatial extent is also calculated for normalization reasons

  19. Temporal • The temporal parameter of the gesture determines the speed of the arm movement of a gesture’s meaning carrying stroke phase and also signifies the duration of movements (e.g., quick versus sustained actions)

  20. Fluidity • Differentiates smooth/graceful from sudden/jerky ones. This concept seeks to capture the continuity between movements, the arms’ trajectory paths as well as the acceleration and deceleration of the limbs • To extract this feature from the input image sequences we calculate the sum of the variance of the norms of the motion vectors

  21. Power/Energy • The power is actually identical with the first derivative of the motion vectors calculated in the first steps

  22. Results of expressivity analysis Spatial Extent EF variation Overall Activation Temporal Fluidity Power/Energy

  23. Expressive synthesis • A system that mimics user’s behaviour through the analysis of facial and gesture signals and expressivity

  24. Synthesis • Greta Platform • BAP calculation • Plane assumption • Inverse kinematics • Manual adaptation • Expressivity features variations implemented in Greta’s BAP interpolation

  25. Synthesis Results

More Related