Derivation of a Learning Rule for Perceptrons

2. Neural Networks Single Layer Perceptrons x1 wk1 x2 wk2  . . . wkm xm Derivation of a Learning Rule for Perceptrons Adaline (Adaptive Linear Element) Widrow [1962] Goal:

3. Neural Networks Single Layer Perceptrons Least Mean Squares (LMS) • The following cost function (error function) should be minimized: i : index of data set, the ith data set j : index of input, the jth input

4. Neural Networks Single Layer Perceptrons Adaline Learning Rule • With then • As already obtained before, Weight Modification Rule • Defining we can write

5. Neural Networks Single Layer Perceptrons Adaline Learning Modes • Batch Learning Mode • Incremental Learning Mode

6. Neural Networks Single Layer Perceptrons Tangent Sigmoid Activation Function x1 wk1 x2 wk2  . . . wkm xm Goal:

7. Neural Networks Single Layer Perceptrons Logarithmic Sigmoid Activation Function x1 wk1 x2 wk2  . . . wkm xm Goal:

8. Neural Networks Single Layer Perceptrons Derivation of Learning Rules • For arbitrary activation function,

9. Neural Networks Single Layer Perceptrons Derivation of Learning Rules Depends on the activation function used

10. Neural Networks Single Layer Perceptrons Derivation of Learning Rules

11. Neural Networks Single Layer Perceptrons Derivation of Learning Rules

12. Neural Networks Single Layer Perceptrons x1 w11  x2 w12 Homework 2 Given a neuron with linear activation function (a=0.5), write an m-file that will calculate the weights w11 and w12 so that the input [x1;x2] can match output y1 the best. • Use initial values w11=1 and w12=1.5, and η= 0.01. • Determine the required number of iterations. • Note: Submit the m-file in hardcopy and softcopy. [x1;x2]=[2;3] [x1;x2]=[[2 1];[3 1]] Case 2 Case 1 [y1]=[5 2] [y1]=[5] • Odd-numbered Student ID • Even-numbered Student ID

13. Neural Networks Multi Layer Perceptrons x1 x2 x3 wlk wji wkj MLP Architecture Hidden layers Input layer Output layer y1 Outputs Inputs y2 • Possessessigmoid activation functionsin the neurons to enable modeling of nonlinearity. • Contains one or more “hidden layers”. • Trained using the “Backpropagation” algorithm.

14. Neural Networks Multi Layer Perceptrons MLP Design Consideration • What activation functions should be used? • How many inputs does the network need? • How many hidden layers does the network need? • How many hidden neurons per hidden layer? • How many outputs should the network have? • There is no standard methodology to determine these values. Even there is some heuristic points, final values are determinate by a trial and error procedure.

15. Neural Networks Multi Layer Perceptrons x1 x2 x3 wlk wji wkj Advantages of MLP • MLP with one hidden layer is a universal approximator. • MLP can approximate any function within any preset accuracy • The conditions: the weights and the biases are appropriately assigned through the use of adequate learning algorithm. • MLP can be applied directly in identification and control of dynamic system with nonlinear relationship between input and output. • MLP delivers the best compromise between number of parameters, structure complexity, and calculation cost.

16. Neural Networks Multi Layer Perceptrons f(.) f(.) f(.) Learning Algorithm of MLP Function signal Error signal • Computations at each neuron j: • Neuron output, yj • Vector of error gradient, ¶E/¶wji Forward propagation “Backpropagation Learning Algorithm” Backward propagation

17. Neural Networks Multi Layer Perceptrons Backpropagation Learning Algorithm If node j is an output node, dj(n) yj(n) netj(n) wji(n) ej(n) yi(n) -1 f(.)

18. Neural Networks Multi Layer Perceptrons Backpropagation Learning Algorithm If node j is a hidden node, dk(n) netk(n) yj(n) yk(n) netj(n) wji(n) wkj(n) yi(n) ek(n) f(.) f(.) -1

19. Neural Networks Multi Layer Perceptrons k j i Right Left k j i Right Left MLP Training • Forward Pass • Fix wji(n) • Compute yj(n) • Backward Pass • Calculate dj(n) • Update weights wji(n+1)