GEOMETRY IN PERCEPTRON LEARNING

1. GEOMETRY IN PERCEPTRON LEARNING Reference: GEOMETRY IN LEARNING, tech. report by KRISTIN P. BENNETT AND ERIN J. BREDENSTEINER, RENSSELAER POLYTECHNIC INSTITUTE IE 5970 SPRING, 2,000

2. PRESENTATION OUTLINE 1. INTRODUCTION 2. A SIMPLE LEARNING MODEL: CONCEPT OF A PERCEPTRON 3. GEOMETRY OF A PERCEPTRON 4. TRAINING: LINEARLY SEPARABLE CASE 4.1. BASIC PROBLEM (PRIMAL-DUAL) 4.2. FIRST SIMPLIFICATION: THE MULTISURFACE METHOD (MSM) 4.2. SECOND SIMPLIFICATION: THE OPTIMAL PLANE 5. TRAINING: LINEARLY INSEPARABLE CASE 5.1. ROBUST LINEAR PROGRAMMING APPROACH (RLP) 5.2. COMBINATIONS OF MSM AND RLP: GENERALIZED OPTIMAL PLANE (GOP) 5.3. COMBINATIONS OF MSM AND RLP: PERTURBED ROBUST LINEAR PROGRAMMING (RLP-P) 6. APPLICATIONS 7. COMPUTATIONAL RESULTS 8. CONCLUSION 9. REMARKS

3. 1. INTRODUCTION • CLASSIFICATION PROBLEM (TWO CLASSES A, B) • IDENTIFY ELEMENTS OF EACH CLASS (EXAMPLE: CANCER DIAGNOSIS, TUMOR BENIGN OR MALIGNANT?) • EACH ELEMENT OF CLASS A OR B IS DESCRIBED BY A VECTOR (N ATTRIBUTES, EXAMPLE: PATIENT’S AGE, BLOOD PRESSURE, SMOKING HABITS) • TRAINING PHASE: CLASS IS KNOWN, FUNCTION F(X) IS CONSTRUCTED • TESTING PHASE: CLASS IS NOT KNOWN, F(X) CLASSIFIES FUTURE POINTS

4. 2. A SIMPLE LEARNING MODEL: CONCEPT OF A PERCEPTRON • PERCEPTRON IS TYPE OF CLASSIFICATION FUNCTION (MOTIVATED BIOLOGICALLY) • PERCEPTRON IS STIMULATED BY (n - DIMENSIONAL) INPUT VECTORS (n ATTRIBUTES, CAN BE SEEN AS COORDINATES)

5. 2. A SIMPLE LEARNING MODEL: CONCEPT OF A PERCEPTRON

6. 3. GEOMETRY OF A PERCEPTRON GEOMETRIC INTERPRETATION:

7. 3. GEOMETRY OF A PERCEPTRON

8. 3. GEOMETRY OF A PERCEPTRON

9. 3. GEOMETRY OF A PERCEPTRON

10. 3. GEOMETRY OF A PERCEPTRON

11. 4. TRAINING: LINEAR SEPARABLE CASE 4.1. BASIC PROBLEM (PRIMAL-DUAL) PRIMAL PROBLEM:

12. 4. TRAINING: LINEAR SEPARABLE CASE 4.1. BASIC PROBLEM (PRIMAL-DUAL) GRAPHICAL INTERPRETATION OF THE PRIMAL PROBLEM:

13. 4. TRAINING: LINEAR SEPARABLE CASE 4.1. BASIC PROBLEM (PRIMAL-DUAL) DUAL PROBLEM:

14. 4. TRAINING: LINEAR SEPARABLE CASE 4.1. BASIC PROBLEM (PRIMAL-DUAL) GRAPHICAL INTERPRETATION OF THE DUAL PROBLEM:

15. 4. TRAINING: LINEAR SEPARABLE CASE 4.2. FIRST SIMPLIFICATION: THE MULTISURFACE METHOD (MSM)

16. 4. TRAINING: LINEAR SEPARABLE CASE 4.2. FIRST SIMPLIFICATION: THE MULTISURFACE METHOD (MSM)

17. 4. TRAINING: LINEAR SEPARABLE CASE 4.2. SECOND SIMPLIFICATION: THE OPTIMAL PLANE

18. 5. TRAINING: LINEARLY INSEPARABLE CASE • General approach: minimize misclassification error • Start from the Multisurface Method (MSM)

19. 5. TRAINING: LINEARLY INSEPARABLE CASE • 5.1. ROBUST LINEAR PROGRAMMING APPROACH (RLP) • minimize the sum of the misclassification errors

20. 5. TRAINING: LINEARLY INSEPARABLE CASE 5.1. ROBUST LINEAR PROGRAMMING APPROACH (RLP)

21. 5. TRAINING: LINEARLY INSEPARABLE CASE 5.2. COMBINATIONS OF MSM AND RLP: GENERALIZED OPTIMAL PLANE (GOP)

22. 5. TRAINING: LINEARLY INSEPARABLE CASE 5.3. COMBINATIONS OF MSM AND RLP: PERTURBED ROBUST LINEAR PROGRAMMING (RLP-P)

23. 6. APPLICATIONS • Determination of heart diseases • Diagnosis of breast cancer • Voting patterns of congressmen to determine party affiliation • Using sonar signals to distinguish between mines and rocks

24. 7. COMPUTATIONAL RESULTS USING MINOS (The sonar data set is completely separable.)

25. 8.CONCLUSION • Perceptron classifies points from two sets • Correct classification only possible in the separable case • Optimization models for training a perceptron were presented • Experiments showed best performance for Generalized Optimal Plane (GOP) and Perturbed Robust Linear Programming (RLP-P)

26. 9. REMARKS • Interesting connection between learning processes and geometry • Limitation to two classes • For RLP-P and GOP the right selection of the parameter lambda • is very important