Support Vector Machines (part 2)

1. Support Vector Machines (part 2) Face Recognition & Biometric Systems

2. Plan of the lecture • SVM – main issues repeated • Soft margin • Multi-class problems • Applications to face recognition • Training set optimization Face Recognition & Biometric Systems

3. SVM – main issues • Aim: data classification • Two stages: • learning (training) • classification Face Recognition & Biometric Systems

4. SVM – main issues • Solves linearly separable problems • Input data are transformed • mapping into higher dimensions • Training: find optimal hyperplane • margin maximisation Face Recognition & Biometric Systems

5. SVM – main issues • A function: • Data mapping: x(x) • Dot product used in all calculations • Dot product -> kernel of convolution • No need to know the function  Face Recognition & Biometric Systems

6. Convolution kernels • Linear • Polynomial • RBF (radial basis functions) Face Recognition & Biometric Systems

7. SVM – main issues • Optimal hyperplane: w0 • x + b0 = 0 • for 2D data it is a line • Optimal margin width: Face Recognition & Biometric Systems

8. SVM – main issues • Optimal hyperplane: • yi – class label • i – Lagrange multipliers (obtained during optimisation) Face Recognition & Biometric Systems

9. SVM – main issues • Lagrange coefficients (): • calculated for every vector from the training set • non-zero for support vectors • equal zero for the majority of vectors • Training set after the optimisation: • support vectors •  coefficients for every vector • number of vectors reduced Face Recognition & Biometric Systems

10. 1 ... n Training Face Recognition & Biometric Systems

11. SVM – main issues • Classification of a vector: xr, xs – support vectors from opposite classes Face Recognition & Biometric Systems

12. Soft margin • Error allowed during the training: • Number of errors minimised • Optimised function must be modified Face Recognition & Biometric Systems

13. Soft margin • Margin maximisation • Minimisation of functional (F – monotonic, convex function): • C – penalty parameter • presentation • Constraints: Face Recognition & Biometric Systems

14. Soft margin • Optimisation without the soft margin: Face Recognition & Biometric Systems

15. Soft margin • Optimisation with the soft margin (for F(u) = u2): Face Recognition & Biometric Systems

16. Multi-class problem • Example Face Recognition & Biometric Systems

17. Multi-class problem • Based on two-class problem • solved by the SVM • N classes in the training set • Possible solutions: • base-class approach • 1 – N comparisons • 1 – 1 comparisons Face Recognition & Biometric Systems

18. The base-class approach • The base-class approach • one class selected as a base class • each class compared with the base class • the strongest response decides • Classification of a single vector: • (N – 1) two-class classifications Face Recognition & Biometric Systems

19. The base-class approach Face Recognition & Biometric Systems

20. The base-class approach Face Recognition & Biometric Systems

21. The base-class approach Face Recognition & Biometric Systems

22. The base-class approach Face Recognition & Biometric Systems

23. The base-class approach Face Recognition & Biometric Systems

24. The base-class approach • Advantages: • high speed • effective when non-base classes are easily separable • Disadvantages: • problems with separating non-base classes Face Recognition & Biometric Systems

25. 1 – N comparisons • Each class compared with the rest • The strongest response decides • Classification of a single vector: • N two-class classifications • Compared to the base-class approach: • more universal (symmetry) • comparable speed Face Recognition & Biometric Systems

26. 1 – N comparisons Face Recognition & Biometric Systems

27. 1 – 1 comparisons • Each class compared with each other • The highest precision • Classification of a single vector: • N(N – 1)/2 two-class classifications • Very slow method Face Recognition & Biometric Systems

28. SVM for face recognition • Detection and verification • Feature vectors comparison • Multi-method fusion • Other applications Face Recognition & Biometric Systems

29. Face detection • Ellipse detection • Generalised Hough Transform • a set of face candidates • Normalisation of the candidates • Verification • image (as a vector) classified by the SVM • multi-class approach Face Recognition & Biometric Systems

30. Feature vectors comparison • Aim: measure similarity between feature vectors • Distance-based similarity: • Euclidean distance • Mahalanobis distance • Similarity measured by the SVM: • two vectors subtracted from each other create a difference vector • difference vector classified K11 K21 K12 K22 ... ... K1n K2n Face Recognition & Biometric Systems

31. Feature vectors comparison K11 - K21 The same class K12 - K22 SVM Different classes ... K1n - K2n Face Recognition & Biometric Systems

32. Feature vectors comparison • Good improvement for EBGM • Eigenfaces not improved • similar results to other metrics Face Recognition & Biometric Systems

33. Multi-method fusion • Many feature extraction methods K1 K1 S1 K2 K2 S2 S ... ... ... Kn Kn Sn Two images Feature vectors Similarities Face Recognition & Biometric Systems

34. Multi-method fusion • Vector of similarities classified • linear kernel • polynomial kernel • time-consuming classification • SVM applied only for the training • linear kernel – weights for the methods (dimensions stand for methods) • average mean based on the weights Face Recognition & Biometric Systems

35. Other applications • Assessment of recognition accuracy • vector of sorted similarities to the elements in the gallery • can be used for many images of the same person • Image quality estimation • e.g. elimination of blurred images Face Recognition & Biometric Systems

36. SVM – limitations • Constant and small number of classes • too much time-consuming for many classes • Training set: • must be representative • optimal number of elements • The parameters must be tuned • Relevance Vector Machines Face Recognition & Biometric Systems

37. Training set optimization • Representative training set: • similarity to the classified data • universal classification rules • difficult to acquire • Real training sets: • data acquired automatically • low quality, faulty data • large number of data Face Recognition & Biometric Systems

38. Training set optimization • Selection of available data • subset drawn randomly • genetic algorithms • Genetic algorithms • heuristic optimization technique • evolutional strategy • population of individuals • fitness • genetic operators: • selection • mutation • crossover Face Recognition & Biometric Systems

39. + – Training set optimization Individual Population drawn Class + Class – Population of training sets SVM training N elements N elements Effectiveness test Effectiveness test Individual Fittness Evolutional operations Face Recognition & Biometric Systems

40. SVM compared to ANN • Support Vector Machines: • more transparent calculations • more controllable than neural networks • implements various types of ANN • useful for image processing • Artificial Neural Networks: • more applications (e.g. compression) • dedicated hardware solutions Face Recognition & Biometric Systems

41. Summary • Soft margin – automatic selection • Multi-class problems: • can be solved basing on two-class problems • various approaches • Many possible applications in the area of face recognition • Training set optimization Face Recognition & Biometric Systems

42. Thank you for your attention! • Next time: Elastic Bunch Graph Matching Face Recognition & Biometric Systems