Support Vector Machine and its Applications Mahnaz Hassibi

1. Support Vector Machine and its Applications Mahnaz Hassibi

2. Outlines • An overview on Support Vector Machines. • The theory behind SVM learning. • It’s differences from other learning systems. • Important issues in instructing SVM model with high performance. • Its application in feature ranking. • Its application in multi-class classification. • Its application in detection and diagnosis.

3. SVM • SVM, Introduced by Vapnik (1995). • SVM is a Statistical Learning Technique. • Classical RegressionAccuracy Depends on Well Defined • PDF. • Not Always, There is Enough Knowledge about • the Probability Distribution of the system. • SVM can perform well even in systems with unknown • distribution. • SVM performs well in High Dimensional Feature Space. • It has a quadratic form and guaranties global minima.

4. Margin Support Vectors Xi. W +b ≥ +1for yi=+1 Xi. W +b ≤ -1 for yi=-1 Margin = 2/|| W|| The Goal is to Maximize the Margin Between the two parallel Hyper-Planes. • Maximize the Margin 2/||W|| • - Minimize ||W||2 With the following constraints:

5. Lagrangian Formulation: For constraints of the form: ,constraint equations are multiplied by positive Lagrange multipliers and subtracted from the objective function Dual Formulation: This is a quadratic optimization problem In which the solution will be the global solution.

6. Kernel Function Non-Linear SVM

7. Kernels: • Kernels map the data into a higher dimension feature space

8. Hyper Parameter Optimization of SVM Kernels • In nonlinear kernels, the parameters defining a kernel are very important in training • process of SVM. • All these methods need to minimize error with respect to hyper parameters. (RBF width, polynomial degree,…) • Empirical selection of kernel’s parameter: A finite number of values • for each parameter is selected and the SVM is trained for them • and values with minimum estimated error builds the kernel. • Non-linear optimization of parameters. • Gradient descent • Conjugate gradient • Newton method • Qausi Newton approach

9. Feature Selection and Ranking with SVM • Smaller feature space. • Reduce computation complexities. • Multivariate methods are prone to over-fitting. • Problems arise when the number of variables is more than number of examples. The objective function comprises two terms: -Accuracy of fit ( maximized). - The number of features (minimized). • Feature ranking and elimination using Linear SVM - SVM performs well in high dimensional space - Because of its linear behavior it can be used for variable ranking and selection - Ranking and feature elimination is based on elements of w -It isa backward feature selection. - In each iteration a feature or group of features are eliminated.

10. SVM-RFE Algorithm for Feature Selection (Non-Linear approach) • The method is based on backward elimination • It starts with all the features and removes one feature at a time • A variable that creates minimum variation of ||w||2 will be removed from the set. • For simplicity and less computational efforts it is assumed that • In this method variable ranking is based on the sensitivity of ||w||2 • with respect to a variable (sensitivity analysis).

11. n=n-1 Eliminating Least Significant Feature Training SVM Data sets with all features Best Selected Features

12. Wrapper Method Using SVM • SVM is used as the predictor model. • SVM model estimates the error of prediction for a selected feature subset. • Leave-one-out procedure can be used to calculate the probability of error . • Because of computational cost of wrapper method, it is difficult to conduct it • for high dimensional systems. • Wrapper is slower that filter method.

13. S1 Feature space sub-set generator S2 Training SVM S3 Data sets with all features Calculate Leave-one-out error Select Best Set

14. Imbalanced Datasets • Over sampling:To replicate the data in the minority class - Increases the training set size and the training time - The probability of over fitting increases • Down sampling: To reduce the size of majority class - It may eliminate some of potentially important data. • SVM Ensembles: To build K-binary SVM by partitioning majority class

15. Majority Class Minority class SVM 1 SVM 2 SVM K Aggregation SVM Ensembles

16. Multi-Class with SVM 1 2 3 4 • One-against All (1-v-r): • One-against One (1-v-1): 1 vs 4 Not 4 • “AND” Gate • Majority vote • Decision Directed Acyclic Graph Not 1 2 3 4 12 3 2 vs 4 1 vs 3 Not 4 Not 1 Not 3 Not 2 34 12 1 vs 2 3 vs 4 23 2 vs 3 4 3 2 1

17. Using SVM for Detection • Fault or abnormality detection. • - Monitoring Machinery in industry. • - Interpretation in medical field. • Anomaly detection. • - Intrusion detection • - misuse detection • - Security issues (internet, network, wireless)

18. Case1- When data sets comprise of both “regular” and “irregular” examples. Standard SVM can be deployed for binary training. • Case2- When examples belong to only one class. • The goal is to identify individuals outside the system’s normal state: • - Available data are only sampled from healthy class. • - There is no sample available to identify outliers. • One-class SVM can be used to detect outliers

19. One-Class SVM • Objective is to find function F( ..), which is positive on subset A • , in feature space, and negative on every thing out side A. • Maximize distance from hyper-plane to Origin. Subset A Hyper Sphere Feature space Hyper plane d Origin is the first member of the negative class

20. Conclusions and Summary • Support Vector Machine is a linear classifier in feature space. • The learning process is based on SRM. • It performs well in high dimensional feature space. • It performs relatively well even in case of small example set vs. high • feature space dimensionality . • In case of non-linear systems, Kernels represent dot product of • projected vectors in feature space. • It has a good application in feature ranking and selection. • It can be deployed for multi-class systems. • It is a powerful tool for classification, diagnosis and detection problems

21. Conclusions and Summary Cont. - Pattern recognition - Signal processing - Data mining - Abnormality detection - Interpretation procedure in different areas of medicine - Intrusion detection ( internet, networking, wireless) - Security issues, identifying different type of Attacks - Information retrieval systems, - Enhancement of search engines results, mail filtering…..