Pattern Recognition: Statistical and Neural

1. Nanjing University of Science & Technology Pattern Recognition:Statistical and Neural Lonnie C. Ludeman Lecture 24 Nov 2, 2005

2. Lecture 24 Topics • Review and Motivation for Link Structure • Present the Functional Link Artificial Neural Network. • Simple Example- design using ANN and FLANN • Performance for Neural Network Designs • 5. Radial Basis Function Neural Networks • 6. Problems, Advantages, Disadvantages, and promise of Artificial Neural Network Design

3. g1(x) g2(x) gj(x) gM(x) Generalized Linear Discriminant Functions w1 x w2 x … wj x + g(x) x … wM x Review 1

4. Patterns are linearly separated in the 3-dim space Separating plane Review 2

5. Example: Decision rule using one nonlinear discriminant function g(x) Given the following g(x) and decision rule Illustrate the decision regions R1 and R2 where we respectively classify as C1 and C2 for the decision rule above Review 3

6. Solution: R1 decide C1 R2 decide C2 every where else every where else Review 4

7. Find a generalized linear discriminant function that separates the classes Solution: d(x) = w1f1(x)+ w2f2(x)+ w3f3(x) + w4f4(x) +w5f5(x) + w6f6(x) = wT f(x) in the f space (linear) Review 5

8. where in the original pattern space: (nonlinear) Review 6

9. Decision Boundary in original pattern space x2 from C1 2 from C2 1 3 4 x1 1 2 -1 Boundary d(x) = 0 -2 Review 7

10. Potential Function Approach – Motivated by electromagnetic theory + from C1 - from C2 Sample space Review 8

11. Plot of Samples from the two classes Review 9

12. Given Samples x from two classes C1 and C2 C C S1 S2 C1 C2 Define Total Potential Function K(x) = ∑ K(x, xk) - ∑ K(x, xk) xk S1 xk S2 Potential Function Decision Boundary K(x) = 0 Review 10

13. Algorithm converged in 1.75 passes through the data to give final discriminant function as Review 11

14. Functional Link Neural Network

15. Quadratic Functional Link

16. Fourier Series Functional Link

17. Principal Component Functional Link fk(x), k=1 to N are chosen as the eigen vectors of the sample covariance matrix

18. Example: Comparison of Neural Net and functional link neural net Given two pattern classes C1 and C2 with the following four patterns and their desired outputs

19. Design an Artificial Neural Network to classify the two patterns given • Design a Functional Link Artificial Neural Network to classify the patterns given. • Compare the Neural Net and Functional Link Neural Net designs

20. (a) Solution: Artificial Neural Net Design Select the following structure

21. After training using the training set and the backpropagation algorithm the design becomes Values determined by neural net

22. (b) Solution: Functional Link Artificial Neural Net Design

23. A neural net was trained using the functional link output patterns as new pattern samples The resulting weights and structure are

24. (c) Comparison Artificial Neural Net (ANN) and Functional Link Artificial Neural Net (FLANN} Designs FLANN has simpler structure than the ANN with only one neural element and Link. Fewer iterations and computations in the training algorithm for FLANN. FLANN design may be more sensitive to errors in patterns.

25. Determining Performance of Neural Net Design on Training Set

26. Test Design on Testing Set Determine Performance for Design using Training Set Classify each member of the training set using the neural network design. Classify each member of the testing set using the neural network design.

27. Could use (a) Performance Measure ETOT (b) The Confusion Matrix (c) Probability of Error (d) Bayes Risk

28. (a) Local and global errors- Used in Neural Net Design procedure Local Measure Global Measure

29. (b) Confusion Matrix- Example Correct Classification Incorrect Classification

30. (c) Probability of Error- Example Estimates of Probabilities of being Correct Estimate of Total Probability of Error

31. (d) Bayes Risk Estimate

32. Radial Basis Function (RBF) Artificial Neural Network Functional Link

33. Functional Form of RBF ANN where Examples of Nonlinearities

34. Design Using RBF ANN Let F(x1, x2, … , xn) represent the function we wish to approximate. For pattern classification F(x) represents the class assignment or desired output (target value) for each pattern vector xa member of the training set Define the performance measure E by E We wish to Minimize E by selecting M,a ,b1, b2, ... , bMandz1, z2, ... zM

35. Finding the Best Approximation using RBF ANN Usually broken into two parts (1st )Find the number M of prototypes and the prototypes { zj : j=1, 2, ... , M} by using a clustering algorithm(Presented in Chapter 6) on the training samples (2nd ) With these fixed M and { zj: j=1,2, ... , M} find the a ,b1, b2, ... , bM that minimize E. Notes: You can use any minimization procedure you wish. Training does not use the Backpropagation Algorithm

36. Problems Using Neural Network Designs Failure to converge Selection of insufficient structure Max iterations too small Lockup occurs Limit cycles Good performance on training set – poor performance on testing set Training set not representative of variation Too strict of a tolerance - “grandmothering”

37. Advantages of Neural Network Designs Can obtain a design for very complicated problems. Parallel structure using identical elements allows hardware or software implementation Structure of Neural Network Design similar for all problems.

38. Other problems that can be solved using Neural Network Designs System Identification Functional Approximation Control Systems Any problem that can be placed in the format of a clearly defined desired output for different given input vectors.

39. Famous Quotation “Neural network designs are the second best way to solve all problems”

40. Famous Quotation “Neural network designs are the second best way to solve all problems” ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

41. Famous Quotation “Neural network designs are the second best way to solve all problems” ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? The promise is that a Neural Network can be used to solve all problems; however, with the caveat that there is always a better way to solve a specific problem.

42. So what is the best way to solve a given problem ???

43. So what is the best way to solve a given problem ??? ?

44. So what is the best way to solve a given problem ??? ? A design that uses and understands the structure of the data !!!

45. Summary Lecture 24 • Reviewed and Motivated Link Structure • Presented the Functional Link Artificial Neural Network. • Presented Simple Example with designs using ANN and FLANN • Described Performance Measures for Neural Network Designs • 5. Presented Radial Basis Function Neural Networks

46. 6. Discussed Problems, Advantages, Disadvantages, and the Promise of Artificial Neural Network Design

47. End of Lecture 24