Maximizing Margin Solutions: Sparse Kernel Classifiers

1. Ch 7. Sparse Kernel MachinesPattern Recognition and Machine Learning, C. M. Bishop, 2006. Summarized by S. Kim Biointelligence Laboratory, Seoul National University http://bi.snu.ac.kr/

2. Contents • Maximum Margin Classifiers • Overlapping Class Distributions • Relation to Logistic Regression • Multiclass SVMs • SVMs for Regression • Relevance Vector Machines • RVM for Regression • Analysis of Sparsity • RVMs for Classification (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

3. Maximum Margin Classifiers • Problem settings • Two-class classification using linear models • Assume that training data set is linearly separable • Support vector machine approaches • The decision boundary is chosen to be the one for which the margin is maximized support vectors (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

4. Maximum Margin Solution • For all data points, • The distance of a point to the decision surface • The maximum margin solution (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

5. Dual Representation • Introducing Lagrange multipliers, • Min. points satisfy the derivatives of L w.r.t. w and b equal 0 • Dual representation Find Appendix E for more details (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

6. Classifying New Data • Optimization subjects to • Found by solving a quadratic programming problem Karush-Kuhn-Tucker (KKT) Conditions  Appendix E : support vectors or • O(N3) (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

7. Example of Separable Data Classification Figure 7.2 (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

8. Overlapping Class Distributions • Allow some misclassified examples  soft margin • Introduce slack variables (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

9. Soft Margin Solution • Minimize • KKT conditions: : trade-off between minimizing training errors and controlling model complexity or (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/ : support vectors

10. Dual Representation • Dual representation • Classifying new data and obtaining b ( hard margin classifiers) (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

11. Alternative Formulation • v-SVM (Schölkopf et al., 2000) - Upper bound on the fraction of margin errors - Lower bound on the fraction of support vectors (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

12. Example of Nonseparable Data Classification (v-SVM) (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

13. Solutions of the QP Problem • Chunking (Vapnik, 1982) • Idea: the value of Lagrangian is unchanged if we remove the rows and columns of the kernel matrix corresponding to Lagrange multipliers that have value zero • Protected conjugate gradients (Burges, 1998) • Decomposition methods (Osuna et al., 1996) • Sequential minimal optimization (Platt, 1999) (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

14. Relation to Logistic Regression (Section 4.3.2) • For data points on the correct side, For the remaining points, : hinge error function (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

15. Relation to Logistic Regression (Cont’d) • From maximum likelihood logistic regression • Error function with a quadratic regularizer (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

16. Comparison of Error Functions Hinge error function Error function for logistic regression Misclassification error Squared error (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

17. Multiclass SVMs • One-versus-the-rest: K separate SVMs • Can lead inconsistent results (Figure 4.2) • Imbalanced training sets • Positive class: +1, negative class: -1/(K-1) (Lee et al., 2001) • An objective function for training all SVMs simultaneously (Weston and Watkins, 1999) • One-versus-one: K(K-1)/2 SVMs • Based on error-correcting output codes (Allwein et al., 2000) • Generalization of the voting scheme of the one-versus-one (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

18. SVMs for Regression • Simple linear regression: minimize • ε-insensitive error function ε-insensitive error function quadratic error function (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

19. SVMs for Regression (Cont’d) • Minimize (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

20. Dual Problem (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

21. Predictions • KKT conditions: (from derivatives of the Lagrangian) (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

22. Alternative Formulation • v-SVM (Schölkopf et al., 2000) fraction of points lying outside the tube (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

23. Example of v-SVM Regression (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

24. Relevance Vector Machines • SVM • Outputs are decisions rather than posterior probabilities • The extension to K>2 classes is problematic • There is a complexity parameter C • Kernel functions are centered on training data points and required to be positive definite • RVM • Bayesian sparse kernel technique • Much sparser models • Faster performance on test data (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

25. RVM for Regression • RVM is a linear form in Chapter 3 with a modified prior (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

26. RVM for Regression (Cont’d) From the result (3.49) for linear regression models • α and β are determined using evidence approximation (type-2 maximum likelihood) (Section 3.5) Maximize (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

27. RVM for Regression (Cont’d) • Two approaches • By derivatives of marginal likelihood • EM algorithm  Section 9.3.4 • Predictive distribution Section 3.3.2 (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

28. Example of RVM Regression v-SVM regression RVM regression • More compact than SVM • Parameters are determined automatically • Require more training time than SVM (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

29. Mechanism for Sparsity only isotropic noise, α = ∞ a finite value of α (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

30. Sparse Solution • Pull out the contribution from αi in Using (C.7), (C.15) in Appendix C (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

31. Sparse Solution (Cont’d) • For log marginal likelihood function L, • Stationary points of the marginal likelihood w.r.t.αi Sparsity: measures the extent to which overlaps with the other basis vectors Quality of : represents a measure of the alignment of the basis vector with the error between t and y-i (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

32. Sequential Sparse Bayesian Learning Algorithm • Initialize • Initialize using , with , with the remaining • Evaluate and for all basis functions • Select a candidate • If ( is already in the model), update • If , add to the model, and evaluate • If , remove from the model, and set • Update • Go to 3 until converged (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

33. RVM for Classification • Probabilistic linear classification model (Chapter 4) • with ARD prior - Initialize - Build a Gaussian approximation to the posterior distribution - Obtain an approximation to the marginal likelihood - Maximize the marginal likelihood (re-estimate ) until converged (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

34. RVM for Classification (Cont’d) • The posterior distribution is obtained by maximizing Iterative reweighted least squares (IRLS) from Section 4.3.3 Resulting Gaussian approximation to the posterior distribution (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

35. RVM for Classification (Cont’d) • Marginal likelihood using Laplace approximation (Section 4.4) • Set the derivative of the marginal likelihood equal to zero, and rearranging then gives • If we define , Same in the regression case (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/

36. Example of RVM Classification (C) 2007, SNU Biointelligence Lab, http://bi.snu.ac.kr/