Learning Larger Margin Machine Locally and Globally

1. Learning Larger Margin Machine Locally and Globally Dept. of Computer Science and Engineering The Chinese University of Hong Kong Shatin, NT. Hong Kong Kaizhu Huang February 9, 2004 The Chinese University of Hong Kong

2. Contributions • Theory: A unified model of Support Vector Machine (SVM), Minimax Probability Machine (MPM), and Linear Discriminant Analysis (LDA). • Practice: A sequential Conic Programming Problem. The Chinese University of Hong Kong

3. Outline • Background And Motivation • Maxi-Min Margin Machine(M4) • Model Definition • Geometrical Interpretation • Solving Methods • Connections With Other Models • M4: Non-separable case • Experimental Results • Future Work • Conclusion The Chinese University of Hong Kong

4. Background: Classifier The Chinese University of Hong Kong

5. SVM A more reasonable decision plane Support Vectors Background: SVM The Chinese University of Hong Kong

6. Maxi-Min Margin Machine(M4) The Chinese University of Hong Kong

7. M4:Geometrical Interpretation The Chinese University of Hong Kong

8. M4:Solving Method • Basic Technique: Divide and Conquer • If we fix to a specific , the problem changes to check whether this satisfies the following constraints: • If yes, we increase ; otherwise, we decrease it. Second Order Cone Programming Problem!!! The Chinese University of Hong Kong

9. M4:Solving Method (Continue) Iterate the following two steps to solve M4: The Chinese University of Hong Kong

10. Yes No can it satisfy the constraints? M4:Solving Method (Continue) The Chinese University of Hong Kong

11. Span all the data points and add them together Connection with MPM Exactly MPM Optimization Problem!!! + The Chinese University of Hong Kong

12. M4 Connection with MPM • Remarks: • The procedure is not reversible: MPM is a special case of M4 • MPM focuses on building decision boundary GLOBALLY, i.e., it exclusively depends on the means and covariances. However, means and covariances may not be accurately estimated. MPM The Chinese University of Hong Kong

13. Connection With SVM SVM with a further assumption: The magnitude of w can scale up without influencing the optimization The Chinese University of Hong Kong

14. M4 M4 M4 SVM SVM SVM Connection With SVM M4 The Chinese University of Hong Kong

15. Connection With SVM SVM assumes The Chinese University of Hong Kong

16. Links With LDA Perform the similar procedure as in MPM LDA The Chinese University of Hong Kong

17. Link With LDA The Chinese University of Hong Kong

18. Non-separable Case The Chinese University of Hong Kong

19. Experimental Results-Synthetic Toy example The Chinese University of Hong Kong

20. Experimental Results-Benchmark Datasets The Chinese University of Hong Kong

21. Future Work • Kernelization? • Nonlinear extension of M4 • Speed-up algorithms? • Is critical in large-scale applications • Generation error bound? • SVM and MPM have both error bounds. • Multi-way classification extension? The Chinese University of Hong Kong

22. Conclusion • Propose a unified model of MPM and SVM • Propose feasible solving methods based on sequential Second Order Cone Programming. The Chinese University of Hong Kong