Model-based Image Interpretation with Application to Facial Expression Recognition

1. Matthias Wimmer matthias.wimmer@cs.tum.edu Model-based Image Interpretation with Application toFacial Expression Recognition

2. Communication Schemes Natural human-computer interaction

3. Example 1: Nissan Pivo2

4. Example 2: Sony’s Smile Shutter

5. Outline of this Presentation facial expression Facial Expression Recognition Model-based image interpretation Adaptive skin color extraction image

6. Facial Expression Recognition

7. What are Facial Expressions? • Six universal facial expressions (Ekman et al.) • Laughing, surprised, afraid, disgusted, sad, angry • Cohn-Kanade-Facial-Expression database • Performed • Exaggerated • Determined by • Shape • Muscle motion

8. Why are they difficult to estimate? • Faces look differently • Hair, beard, skin-color, … • Different facial poses • Only slight muscle activity

9. Our Approach motion features and structural features

10. Model Fitting with Learned Objective Functions

11. Model-based image interpretation • The model The model contains a parameter vector that represents the model’s configuration. • The objective functionCalculates a value that indicates how accurately a parameterized model matches an image. • The fitting algorithmSearches for the model parameters that describe the image best, i.e. it minimizes the objective function.

12. Local Objective Functions

13. Ideal Objective Functions P1: Correctness property:Global minimum corresponds to the best fit. P2: Uni-modality property:The objective function has no local extrema. ¬ P1 P1 ¬P2 P2 • Don’t exist for real-world images • Only for annotated images: fn( I , x ) = | cn – x |

14. Learning the Objective Function • Ideal objective function generates training data • Machine Learning technique generates calculation rules x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

15. Benefits of the Machine Learning Approach • Accurate and robust calculation rules • Locally customized calculation rules • Generalization from many images • Simple job for the designer • Critical decisions are automated • No domain-dependent knowledge required • No loops

16. Evaluation: Fitting Accuracy

17. Adaptive Skin Color Classification

18. Basics about Skin Color Classification • Skin color depends on several image conditions • Skin color occupies a large cluster • Skin color varies greatly within a set of images. • Skin color varies slightly within one image. image 1 image 2 green green image 2 red red

19. Our Approach • Learn image-specific skin color characteristics • Parameterize a skin color classifier accordingly Offline: • Learn the skin color mask • Specific for the face detector Online: • Detect the image specific skin color model • Using the face detector • Using the skin color mask • Adapt skin color classifier

20. Results • Robustness: • Detection of facial parts:eyes, lips, brows,… • Exact shape outline • Ethnic groups original image fixed classifier adapted classifier Correctly detected pixels: • fixed classifier: 90.4% 74.8% 40.2% • adapted classifier: 97.5% 87.5% 97.0% • improvement: 1.08 1.17 2.41

21. Additional Work lip classifier eye brow classifier iris classifier tooth classifier

22. Conclusion and Outlook

23. Conclusion • Possible to derive information from face images • Model-based image interpretation is beneficial • Learn crucial decisions within algorithms • Don’t specify parameters by trial and error • Adaptive skin color classifier • Learned objective functions • Not yet reached goal for natural HCI • Progress is clearly visible. → Goal is achievable!

24. Outlook • Learn global objective function • Learn discriminative function (direct parameter update) • Rendered AAM provides training images • Many images • Exact ground truth (no manual work required) • Learn with further features • Higher number of features • SIFT, LBP, … • Learn with better classifiers • Relevance Vector Machines • Boosted regressors

25. Thank you! Online-Demonstration: http://www9.cs.tum.edu/people/wimmerm

26. Publications 2008 • Tailoring Model-based Techniques for Facial Expression Interpretation. ACHI08 • Face Model Fitting with Generic, Group-specific, and Person-specific Objective Functions. VISAPP • Low-level Fusion of Audio and Video Feature for Multi-modal Emotion Recognition. VISAPP • Facial Expression Recognition for Human-robot Interaction - A Prototype. Robot Vision 2007 • Audiovisual Behavior Modeling by Combined Feature Spaces. ICASSP • Emotionale Aspekte in Produktevaluationen. Multimediatechnik • Application of emotion recognition methods in automotive research. Emotion and Computing • Human Capabilities on Video-based Facial Expression Recognition. Emotion and Computing • SIPBILD - Mimik- und Gestikerkennung in der Mensch-Maschine-Schnittstelle. INFORMATIK • Learning Robust Objective Functions with Application to Face Model Fitting. DAGM • Automatically Learning the Objective Function for Model Fitting. MIRU • Initial Pose Estimation for 3D Models Using Learned Objective Functions. ACCV • Estimating Natural Activity by Fitting 3D Models via Learned Objective Functions. VMV • Learning Local Objective Functions for Robust Face Model Fitting. PAMI (journal paper) • Enabling Users to Guide the Design of Robust Model Fitting Algorithms. ICV 2006 • Learning Robust Objective Functions for Model Fitting in Image Understanding Applications. BMVC • A Person and Context Specific Approach for Skin Color Classification. ICPR • Adaptive Skin Color Classificator. Journal on Graphics, Vision and Image Processing (journal paper) • Bitte recht freundlich. Zukunft im Brennpunkt (journal paper) 2005 • Sensor-based Situated, Individualized, and Personalized Interaction in Smart Environments. INFORMATIK • Adaptive Skin Color Classificator. GVIP 2004 • Experiences with an Emotional Sales Agent. Affective Dialogue Systems