Pattern Recognition: Statistical and Neural

1. Nanjing University of Science & Technology Pattern Recognition:Statistical and Neural Lonnie C. Ludeman Lecture 18 Oct 21, 2005

2. Lecture 18 Topics 1. Example – Generalized Linear Discriminant Function 2. Weight Space 3. Potential Function Approach- 2 class case 4. Potential Function Example- 2 class case 5. Potential Function Algorithm – M class case

3. Classes not Linearly separable from C1 from C2 2 1 3 4 x1 1 2 -1 -2 Q. How can we find decision boundaries?? Answers: (1) Use Generalized Linear Discriminant functions (2) Use Nonlinear Discriminant Functions

4. Example: Generalized Linear Discriminant Functions x2 from C1 from C2 3 2 1 3 4 x1 1 2 -1 -2 Given Samples from 2 Classes

5. Find a generalized linear discriminant function that separates the classes Solution: d(x) = w1f1(x)+ w2f2(x)+ w3f3(x) + w4f4(x) +w5f5(x) + w6f6(x) = wT f(x) in the f space (linear)

6. where in the original pattern space: (nonlinear)

7. Use the Perceptron Algorithm in the f space (the iterations follow) Iteration # Samples Action Weights

8. d(x) Iterations Continue Iterations Stop

9. The discriminant function is as follows Decision boundary set d(x) = 0 Putting in standard form we get the decision boundary as the following ellipse

10. Decision Boundary in original pattern space x2 from C1 2 from C2 1 3 4 x1 1 2 -1 Boundary d(x) = 0 -2

11. Weight Space To separate two pattern classes C1 and C2 by a hyperplane we must satisfy the following conditions Where wTx = 0 specifies the boundary between the classes

12. But we know that wTx = xTw Thus we could now write the equations in the w space with coefficients representing the samples as follows Each inequality gives a hyperplane boundary in the weight space such that weights on the positive side would satisfy the inequality

13. In the Weight Space

14. View of the Pereptron algorithm in the weight space

15. Potential Function Approach – Motivated by electromagnetic theory + from C1 - from C2 Sample space

16. Given Samples x from two classes C1 and C2 C C S1 S2 C1 C2 Define Total Potential Function K(x) = ∑ K(x, xk) - ∑ K(x, xk) xk S1 xk S2 Potential Function Decision Boundary K(x) = 0

17. Choices for Potential functions K(x, xk)

18. Graphs of Potential functions

22. Example – Using Potential functions Given the following Patterns from two classes Find a nonlinear Discriminant function using potential functions that separate the classes

23. Plot of Samples from the two classes

24. Trace of Iterations

25. Algorithm converged in 1.75 passes through the data to give final discriminant function as

26. KFINAL(x)

27. x1

28. Potential Function Algorithm for K Classes Reference (3) Tou And Gonzales

31. Flow Chart for Potential Function Method: M-Class

32. Flow Chart Continued

33. Flow Chart Continued

34. Summary 1. Example – Generalized Linear Discriminant Function 2. Weight Space 3. Potential Function Approach- 2 class case 4. Potential Function Example- 2 class case 5. Potential Function Algorithm – M class case

35. End of Lecture 18