1 / 41

3D Models for Face Image and Video Processing

3D Models for Face Image and Video Processing. Gábor Szirtes ELTE Dept of Information Systems Neural Information Processing Group Lőrincz-lab. Content. Short Introduction Motivations Pile of concepts ( framework ?) Future applications Ongoing projects and immature ideas What next?.

claire
Download Presentation

3D Models for Face Image and Video Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. 3D Models for Face Image and Video Processing Gábor Szirtes ELTE Dept of Information Systems Neural Information Processing Group Lőrincz-lab SSIP 2002 Budapest, Hungary

  2. Content • Short Introduction • Motivations • Pile of concepts(framework?) • Future applications • Ongoing projects and immature ideas • What next? SSIP 2002 Budapest, Hungary

  3. A few words about our group • Since 1999 • Within the Information Systems Department, ELTE • 5 PhD students and ~20 grad students • Mainly biologically motivated projects • RL, ICA, machine learning, facial expressions, dynamical systems, image processing SSIP 2002 Budapest, Hungary

  4. Motivations • Being in quest of the ‘Holy Grail’: intelligence • One working example: our brain • Evolutionary concepts, need for adaptation • Perception and Action SSIP 2002 Budapest, Hungary

  5. Conceptual framework I. ? SYSTEM Perception Action ENVIRONMENT Noisy, stochastic, evolving SSIP 2002 Budapest, Hungary

  6. Conceptual framework II. Central hypothesis • Internal representation (encoded signals from the environment and the system’s state) • Reconstruction SSIP 2002 Budapest, Hungary

  7. Perception • Active • Not simply feed-forward • Feed-back modulated and controlled • Modular • Component based • Adaptive, ‘plastic’ SSIP 2002 Budapest, Hungary

  8. Perception II. • Active: it is not a passive signal detection process. We need to `foresee` and anticipate the expected changes (prediction). • Influenced by higher order modulation (e.g. FOA, focus of attention, conscious and unconscious perception) SSIP 2002 Budapest, Hungary

  9. Perception III. • Several stages of processing • Not purely hierarchical (feed-back) • Distributed, parallel ways, strong interplay Modularity SSIP 2002 Budapest, Hungary

  10. Perception IV. Components: meaningful (?) building blocks SSIP 2002 Budapest, Hungary

  11. Perception V. This is what we have seen before? SSIP 2002 Budapest, Hungary

  12. Parts… Drawings of 4 year old healthy children SSIP 2002 Budapest, Hungary

  13. …and the whole Drawing of a 3 and a half year old child with autism SSIP 2002 Budapest, Hungary

  14. Beyond the theory Recognition of faces and facial expressions • Twofold goals: • Understand perception • Help develop applications for Human-Computer Interaction Psychiatric analysis and treatment … SSIP 2002 Budapest, Hungary

  15. Database In collaboration with the Psychiatric Clinic of SOTE (Simon-lab) SSIP 2002 Budapest, Hungary

  16. A few examples of segmented images Happiness Disgust SSIP 2002 Budapest, Hungary

  17. The architecture ACTION ? RL container M* M** M1 M2 M3 FACES SSIP 2002 Budapest, Hungary

  18. Modules for recognition of faces • Finding heads: Skin detection • Tracking: particle filtering • Segmentation • 3D model based transformation • Identification, recognition or analysis • (back-transformation) SSIP 2002 Budapest, Hungary

  19. Module 1 Face location (fitting) • Many heuristics are possible • One particular choice skin-detector SSIP 2002 Budapest, Hungary

  20. Skin detector b g Skin color cluster learned by MLP r SSIP 2002 Budapest, Hungary

  21. Module 2+ Particle filtering Segmentation Tracking CONDENSATION (Conditional Density Propagation ) (Isard and Blake, 1998) SSIP 2002 Budapest, Hungary

  22. Segmentation Image based Feature based procedures • Two approaches: • approximating contours with splines or snakes (too many degrees of freedom) • Template based A simple template SSIP 2002 Budapest, Hungary

  23. Segmentation II. More sophisticated manually tuned template Arbitrary spine directions (with positive-negative weights) SSIP 2002 Budapest, Hungary

  24. Segmentation III. Many concurrent candidates SSIP 2002 Budapest, Hungary

  25. Segmentation IV. Head-shoulder template for better fitting SSIP 2002 Budapest, Hungary

  26. Particle filtering in action! Initialization made by hand SSIP 2002 Budapest, Hungary

  27. Well, there is no perfect method… Sometimes even the best choice is far from the face to be tracked… SSIP 2002 Budapest, Hungary

  28. Tracking of fast motion against a cluttered background From http://www.robots.ox.ac.uk/~misard/condensation.html SSIP 2002 Budapest, Hungary

  29. CONDENSATION Keywords: general, multi-modal densities, sampling, Discrete -continuous Markovian SSIP 2002 Budapest, Hungary

  30. Module 3 (off the stream) Facial expression (display) recognition SSIP 2002 Budapest, Hungary

  31. Facial expression recognition surprise HMM winner: surprise HMM on segmented image sequences Reconstruction error HMM emission SSIP 2002 Budapest, Hungary

  32. Module 4 3D face model Extension of the CANDIDE (Rydfalk,1987) model Compatible with FACS (Ekman and Friesen, 1977) Candide 3 (developed for MPEG4 standard) SSIP 2002 Budapest, Hungary

  33. How to use the model? Target (synthetic) face Searching SSIP 2002 Budapest, Hungary

  34. Such a big space! • Reconstruction error based optimization problem • Too many local minima • Global optimum finding procedure: STAGE (Boyan, 1998) SSIP 2002 Budapest, Hungary

  35. STAGE • Algorithm for finding the global optimum • Function approximator • learns an evaluation function that predicts the outcome of a local search • Experience: it is able to explore the global structure Let us find the minimum of F(x)=(|x|-10)cos(2x) SSIP 2002 Budapest, Hungary

  36. STAGE II • It can be combined with any local search method (hillclimbing,WALKSAT,…) • It works on both the objective and the evaluation function at two stages • Smart restart by a better prediction • Real-valued (compared to GA) • Easy to implement SSIP 2002 Budapest, Hungary

  37. SSIP 2002 Budapest, Hungary

  38. What we have got so far? • A few working modules • Working RL architectures • Working combination in an other problem domain: Internet search • …and research is focused on how to link all of our concepts. SSIP 2002 Budapest, Hungary

  39. What next? • Many avenues • Ongoing projects with psychiatrists: trajectory analysis with cliplets, transient expressions, depression quantification… • Distance learning • Human Computer Interaction • Virtual reality SSIP 2002 Budapest, Hungary

  40. Infos about our research activityhttp://people.inf.elte.hu/lorincz/ SSIP 2002 Budapest, Hungary

  41. Thanks for your attention (and patience)! SSIP 2002 Budapest, Hungary

More Related