Analysis of Robust Soft Learning Vector Quantization

1. Analysis of Robust Soft Learning VectorQuantization Gert-Jan de Vries (gj.de.vries@philips.com) February 2009

2. Outline • Learning Vector Quantization • Robust Soft LVQ • Analyses in controlled environment • Simulations • Mathematical • Application on real world data • Summary

3. N-dim Learning Vector Quantization (LVQ) • Training: • Initialize prototypes • Present data sample • Select closest prototype(s) • Update prototype(s) • Attract • Repel • Classification: • Voronoi tessellation

4. LVQ update scheme • Prototype w of class S • Update at time step μ • Present sample ξ of class σ • Learning rate η • Update strength hS • Distances {dT = dist(ξ,wT)} • Data class label σ • … • - e.g. LVQ2.1: • Along line through ξ, wS

5. Robust Soft LVQ (RSLVQ)* • Prototype update • Closest correct • Closest incorrect • Assignment probabilities: * [Seo & Obermayer. 2003]

6. Robust Soft LVQ (RSLVQ) • Assumption on distribution around prototypes (Gaussian) • Controlled environment* with 2 prototypes (2 classes) • Distance-based • Further away, larger update (basic LVQ) • Relative distance (information gain) • Takes care of stability issues * [Biehl et al. 2007]

7. Mathematical Analysis • Controlled environment • 7 characteristic quantities • Describe development of characteristic quantities in terms of recurrence relations • Use prototype update formula • Rewrite recurrence relations as differential equations (ODE) • Coupled system of 7 ODE’s • Perform averages on the ODE’s • Use characteristics of data model

8. Analysis (2): Φ-correction • Update function: • Replace by: • Slope correction • Equal slope at d=0 • Analytical solution ODEs in limit N∞, η0

9. Comparison of ODE’s and simulation Original Φ-corrected • Evolution of characteristic quantities • Simulations and ODE solutions coincide for both ‘original’ and ‘Φ-corrected’ RSLVQ

10. Comparison of ODE’s and simulation (2) • Slight differences between ‘original’ and ‘Φ-corrected’, however similar tendencies • Equal generalization error for ‘original’ and ‘Φ-corrected’ RSLVQ.

11. Analysis: Effect of softness parameter Eg (large #epochs) Eg (asymptotic) vsoft vsoft Analytical Simulations

12. Facial Expressions (real world data set) * [Kanade et al. 2000] 108x147 pixel images*

13. Facial Expressions: Feature Extraction (LBP) * [Ojala et al. 2002] • Per grey valued (ic) pixel, calculate Local Binary Pattern (LBP): • Only consider LBP with at most 2 bitwise transitions* (yields L=59) • These ‘uniform patterns’ represent local primitives such as edges and corners

14. Facial Expressions: Feature Extraction (LBP) • Divide image into (42) regions • Per region Rj obtain histogram of LBP values: • Combine histograms into a single vector of length N = 42*59 = 2478 • Using overlapping regions (overlap=1/2): 143 regions  N = 8437

15. Facial Expressions: Classification • Facial expressions of 95 university students • 7 classes: Anger, Disgust, Fear, Joy, Sadness, Surprise, Neutral • 1214 instances • 10-fold cross validation • Testing with unseen people

16. Facial Expressions: Training 92.2% 93.2% Correct rate Correct rate vsoft vsoft LBP LBP with overlap Similar influence of softness on performance as in mathematical analysis (with limited learning time) Sometimes numerical problems

17. Facial Expressions: Training 93.6% <Images to be included> Correct rate All Single All double Number of prototypes per class Number of prototypes per class Using optimal parameter settings as single prototype case

18. Facial Expressions: Training Mean and standard deviation over 10 fold cross validation * [Gritti et al. 2008] Results comparable with SVM

19. Summary • Analytical description of learning dynamics of RSLVQ • Within controlled environment • Analysis for N∞ • (Asymptotic) generalization error studied • Application to facial expression data • Similar effects on softness parameter • No large effect on number of prototypes per class • RSLVQ performs well

20. Acknowledgements References M. Biehl, A. Gosh, and B. Hammer. Dynamics and generalization ability of LVQ algorithms. In Journal of Machine Learning Research, 8 (Feb):323-360, 2007. T. Kanade, J. Cohn, and Y. Tian. Comprehensive database for facial expression analysis. In IEEE International Conference on Automatic Face & Gesture Recognition (FG), 46-53, 2000. T. Ojala, M. Pietikäinen, and T. Mäenpää. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):971-987, 2002. S. Seo, and K. Obermayer, Soft learning vector quantization, Neural computation, 15:1589-1603, 2003 C. Shan, S. Gong, and P.W. McOwan. Robust facial expression recognition using local binary patterns. In IEEE International Conference on Image Processing (ICIP), volume 2:370-373, Genoa, Italy, September 2005. T. Gritti, C. Shan, V. Jeanne, R. Braspenning, Local Features based Facial Expression Recognition with Face Registration Errors, FG 2008. • Philips Research – Video Processing & Analysis: • Vincent Jeanne • Caifeng Shan • Rijksuniversiteit Groningen: • Michael Biehl