Computation in neural networks

1. Computation in neural networks M. Meeter

2. Calculating a function Perceptron learning problem Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1] [+1, +1, -1] [-1, -1, -1, -1] [-1, -1, +1, -1] [-1, -1, -1] [-1, +1, +1, -1] [-1, +1, +1] [+1, -1, +1, -1]

3. Types of networks & functions • Attractor • Feedfwrd Hebbian • associative (Hebbian) • competitive • Feedfwrd error corr. • perceptron • backprop • completion, autoass. memory • association, assoc. memory • clustering • categorization, generalization • nonlinear, same

4. Types of networks • Attractor • Feedfwrd Hebbian • associative (Hebbian) • competitive • Feedfwrd error corr. • perceptron • backprop • completion, autoass. memory • association, assoc. memory • clustering • categorization, generalization • nonlinear, same

5. A Classification

6. Generalization 76 128 ?

7. Regression = generalization Univariate Linear Regression prediction of values

8. Clustering

9. Types of networks • Attractor • Feedfwrd Hebbian • associative (Hebbian) • competitive • Feedfwrd error corr. • perceptron • backprop • completion, autoass. memory • association, assoc. memory • clustering • categorization, generalization • nonlinear, same

10. Classification - discrete Perceptron learning problem Prototypical Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1] [+1, +1, -1] [-1, -1, -1, -1] [-1, -1, +1, -1] [-1, -1, -1] [-1, +1, +1, -1] [-1, +1, +1] [+1, -1, +1, -1]

11. Classification - discrete Perceptron learning problem Prototypical Input Patterns Desired output [+1, +1, -1, -1] [+1, -1, +1] [-1, -1, +1, +1] [+1, +1, -1] [-1, -1, -1, -1] [-1, -1, +1, -1] [-1, -1, -1] [-1, +1, +1, -1] [-1, +1, +1] [+1, -1, +1, -1]

12. Xi X1 X2 Xn Classification in Perceptron  wji threshold

13. Effe tussendoor… • Bij perceptron etc.: net input knoop>0 dan activatie 0 • Niet altijd gewenst: daarom heeft knoop in continue vormen perceptron / backprop een ‘bias’, een activatie die altijd bij input opgeteld wordt • Effect: verschuiven threshold

14. - Threshold Input= + Input= mixture Threshold Classification in 2 dimensions

15. Discriminant Analysis Find center of two categories, draw line in between, then one diagonal in middle = discrimination line Produces exact same result

16. Generalization = Regression Univariate Linear Regression prediction of values

17. Activation function  Xi (·) X1 X2 v =  xi*wji (v) = av + b Xn   Perceptron with linear activation rule y wji j Change weights with  rule, minimizing Se2 Bias

18. X 1 X 2 X i X n Multivariate Multiple Linear Regression Multivariate = multiple independent variables X =multiple inputs y 1 1 y 2 2 Multiple = multiple dependent variables Y =multiple outputs Y2 X Y1

19. linear nonlinear y y x x Linear vs. nonlinear regression • Here: quadratic • General: wrinkle-fitting

20. X X Multi-Layer Perceptron å = v x * w i ji i  y1 y2 • Fit any function: “Universal approximators” X= [x1, x2, .., xi, .., xn]

21. Too complex model y x Bad Extremely bad Overfitting y Too simple model x

22. Clustering Competitive learning: • next week • ART

23. Conclusions • Neural networks similar to statistical analyses • Perceptron -> categorization / generalization • Backprop -> same but nonlinear • Competitive l. -> clustering • But… • Whole data set vs. one pattern at a time

24. Feature reduction with PCA

25. ? ? Feature extraction with PCA Unsupervised Learning Hebbian Learning