Kernel Methods

1. Kernel Methods A B M Shawkat Ali

2. Data Mining ¤DM or KDD (Knowledge Discovery in Databases) Extracting previously unknown, valid, and actionable information crucial decisions ¤Approach Model crucial decisions Train Data Test Data

3. History of SVM • The original optimal hyperplane algorithm proposed by Vladimir Vapnik in 1963 was a linear classifier. • However, in 1992, Bernhard Boser, Isabelle Guyon and Vapnik suggested a way to create non-linear classifiers by applying the kernel trick (originally proposed by Aizerman et al.) to maximum-margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a non-linear kernel function. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. The transformation may be non-linear and the transformed space high dimensional; thus though the classifier is a hyperplane in the high-dimensional feature space, it may be non-linear in the original input space.

4. Property of the SVM ¤Relatively new approach ¤Lot of interest recently:  Many successes, e.g., text classification ¤Important concepts: • Transformation into high dimensional space  Finding a "maximal margin" separation • Structural risk minimization rather than Empirical risk minimization

5. Support Vector Machine (SVM) ¤Classification  Grouping of similar data. ¤Regression Prediction by historical knowledge. ¤Novelty Detection  To detect abnormal instances from a dataset. ¤Clustering, Feature Selection

6. SVM Block Diagram                Non linear Mapping by Kernel Training Data Domain            To Choose Optimal Hyperplane                           Linear Feature Space of SVM

7.         Class I          Class II SVM Block Diagram Constructed Model through Feature knowledge Kernel Mapping                 Test Data Domain

8. SVM Formulation

9. SVM Formulation

10. Mercer’s Condition SVM Formulation

11. Types of Kernels Common kernels for SVM ¤ Linear ¤ Polynomial ¤ Radial Basis Function New kernels (not used in SVM) ¤Laplace ¤Multiquadratic

12. SVM kernel • Linear Polynomial • Gaussian (Radial Basis Function)

13. Laplace kernel Introduced by Pavel Paclik et. al. in Pattern Recognition letters 21 (2000) Laplace Kernel based on Laplace Probability Density Smoothing Parameter (Sp)

14. Linear Kernel

15. The reality of data separation

16. RBF kernel

17. XOR solved by SVM Table 5.3. Boolean XOR Problem

18. First, we transform the dataset by polynomial • kernel as: Here,

19. Therefore the kernel matrix is: , We can write the maximization term following SVM implementation given in Figure 5.20 as: subject to:

20. . By solving these above equations we can writethe solutionto this optimisationproblem as: Therefore, the decision function in the inner product representation is:

21. The 2nd degree polynomial kernel function: Now we can write the 2nd degree polynomial transformation function as:

23. = Therefore the optimal hyperplane function for this XOR problem is:

24. Conclusions • Research Issues • How to select a kernel automatically • How to select optimal parameter values for kernel