Reactive Behavior Modeling Neural Networks (GATE-561)

1. Reactive Behavior ModelingNeural Networks(GATE-561) Dr.Çağatay ÜNDEĞER Instructor Middle East Technical University, GameTechnologies Bilkent University, Computer Engineering & General Manager SimBT Inc. e-mail : cagatay@undeger.com Game Technologies Program – Middle East Technical University – Fall 2009

2. Outline • Neural Networks

3. Artificial Neural Networks (NNets) • Inspired by human brain. • Fundamental functional units of brain are: • Neurons / soma / nerve cells and • Axons / dendrites connecting nerve cells.

4. Artificial Neural Networks (NNets) • Signals can be thought as simple electrical impulses from axons to dendrites. • Dendrites collect the signals. • Soma performs a kind of summation. • Axon fire and transmit the signal.

5. Artificial Neural Networks (NNets) • Neural networks use a similar approach. • Consist of: • Input arcs collect the signals, • Neuron sums the signals and • Output arcs transmit the summed signal based on a transmit function. • Neural networks can be learned.

6. Neural Network Architecture Arcs (storing weights) Neurons h h i input 1 h h o output 1 i input 2 h h o output 2 i h h input 3 input layer hidden layers output layer

7. Usage of NNets • Environmental Scanning and Classification: • Interpret vision and auditory information • Memory: • Can learn through experience • Behavior Control: • Outputs can be used as actions to be performed. • Response Mapping: • Any input output mapping and association.

8. Usage of NNets • The usage of NNets is a little bit fuzzy. • No exact cases and rules. • Assume it as a tool and use your creativity to get whatever you like. • What NNets do is • To seperate a solution space into partitions

9. B=1 X1 X2 Xn Xi A Neuron Architecture B = a constant bias node, which may represent past history w1 b w2 y to input of other neurons Y wi Xi = input i of neuron wi = weight of input i wn Y = summing junction y = output of neuron y is based on a transfer or activation function

10. B=1 X1 X2 Xn Xi n Ya = B*b + ΣXi*wi i=1 Neuron Architecture Summing is referred as the input activation Ya w1 b w2 y Y wi Input activation is fed into the activation function fa(x) and output y is acquired wn Inputs and weights can be in range (-∞, +∞) y = fa(x)

11. Some Activation Functions y y 1 1 0 0 x x θ Step function Linear function fs(x) = 1, if x≥θ 0, if x<θ fs(x) = x

12. Common Activation Function y θ = 1 1 0 x Exponential / Sigmoid function fs(x) = 1 / ( 1 + e-θx ) You can also scale and shift the output if required

13. A Group Of Neurons • A single neuron will not do a lot for us • We need a group of them A two layer NNet A single layer NNet

14. Simple Logic Functions • Created by McCulloch and Pitts in 1943. • AND, OR, XOR functions could be done with simple neural network structures.

15. Simple Logic Function AND OR XOR

16. Learning NNets • Weights of NNets can be learned by various algorithms. • The most common algorşthm is the Backpropagation.

17. Backpropagation • Can learn weights of a multi-layer network • With fixed set of units and interconnections • Employs a gradient descent search • To minimize the squared errors between target output and network output • It is sub-optimal and may fail to reach true optimal point.

18. Backpropagation • Can learn weights of a multi-layer network • With fixed set of units and interconnections • Employs a gradient descent search • To minimize the squared errors between target output and network output

19. Backpropagation • Error surface of the network may have multiple local minima. • Therefore backprogation guarantees only to converge a local-optimum and may fail to reach the true-optimum.

20. Backpropagation 11 01 input 1 21 12 output 1 02 input 2 13 22 output 2 03 14 input 3 Initialize the network weights with small random numbers (e.g. -0.5 ... 0.5) Train the network with m training examples repeatedly for multiple iterations. Stop training when a convergence criteria is met.

21. Backpropagation 11 01 actual expected x01 21 12 t1 o1 02 x02 13 22 t2 o2 03 14 x03 For one training example, assume that: Inputs are x01, x02 and x03 and Expected outputs are t1 and t2 We propogate the inputs through the network and Compute the actual outputs o1, o2 and o3

22. Backpropagation 11 01 actual expected x01 21 12 t1 o1 δ21 02 x02 13 22 t2 o2 δ22 03 14 x03 output errors Errors will occur between actual and expected outputs. We propagate errors backward through the network. Compute error on each output unit k ( δk ) using: δk = ok (1 - ok) (tk - ok) e.g. δ21 = o21 (1 - o21) (t21 - o21)

23. Backpropagation δ11 11 w11-21 01 actual expected x01 w11-22 δ12 21 12 t1 o1 δ21 02 x02 δ13 13 22 t2 o2 δ14 δ22 03 14 x03 hidden unit errors output errors e.g. δ11 = o11 (1 - o11) (w11-21δ21 + w11-22δ22 ) Compute error on each hidden unit h (δh) using: δh = oh (1 - oh) Σ wh-kδk

24. Backpropagation x01 δ11 w01-11 11 w11-21 01 actual expected w01-12 x01 w11-22 δ12 21 12 t1 o1 δ21 02 x02 δ13 13 22 t2 o2 δ14 δ22 03 14 x03 hidden unit errors output errors e.g. w01-11 = w01-11 + ηδ11 x01-11 η = learning rate (e.g. 0.05) Update each weight w (wf-t) using: wf-t = wf-t + ηδt xf-t

25. Backpropagation • The main loop of the algorithm should repeatedly iterate over the training examples • May iterate thousands of times using the same training examples. • Until the network performs acceptably well: • Test with the entire training set or • A better way: test with a sparate test set.

26. Enemy Dead Sound Low Health Wander Attack Spawn Retreat Search An Action SelectorNeural Network • The inputs are state variables that equal to either 0 (no) or 1 (yes). • Outputs are action variables that are between 0 and 1. • The action with the greatest output is selected.

27. A Number Recognizer Neural Network • Problem : To recognize normalized bitmaps of handwritten digits from “0” to “9”. • Data Set : Collected from 43 people • 30 people to training set • 13 people to test set • 32x32 bitmaps are divided into 4x4 blocks and black pixels are counted • So we have 8x8 matrices and each member stores a number between 0 and 16

28. A Number Recognizer Neural Network • 8x8 pixels = 64 inputs • No Missing Input Value • Target Value : class number from 0 to 9 • Training Set : 3823 Instances • Test Set : 1797 Instances

29. A Number Recognizer Neural Network Bias Control Unit x012 = 1 x013 = 1 x011 = 1 x112 x113 “0” x111 1 1 1 . . . . . x212 x213 . . . . . . . . . . . . x6412 Max xn13 x0641 = 1 x1641 64 n 10 “9” Result 64 Pixel values (0..16) of 8x8 image

30. A Number Recognizer Neural Network • For each optic character (0..9), an output unit is used • The class of the output unit having the highest value is chosen as result. • Sample target values for training:

31. A Number Recognizer Neural Network • Random weights are given (–0.5 ... 0.5) • A fixed number of iterations are performed on Training Set • Meanwhile weights of iterations are saved • A test phase is performed using saved weights on Test Set to avoid overfitting • The iteration weights that maximizes the test set accuracy is chosen

32. Test Platform General Information New Instance Real Value Result of NNet Output layer of NNet Loaded Instance Set Current instance

33. First Attempt • 4 hidden units • 0.1 as learning rate • 5000 iterations are performed (>12 hours) • Best on Test Set = 86.9% on 450th iteration • Meanwhile 92.2% on Training Set • 82.5% accuracy on 5000th iteration • Best on Training Set = 93.0%on 1110th iteration • Meanwhile 86.1% on Test Set

34. Graph of First Attempt

35. Second Attempt • 4 hidden units to 8 • 0.1 as learning rate (same) • 5000 iterations to 2500 • Best on Test Set = 94.0% on 100th iteration • Meanwhile 97.7% on Training Set • 88.9% accuracy on 2500th iteration • Best on Training Set = 98.0%on 140th iteration • Meanwhile 93.6% on Test Set

36. Graph of Second Attempt

37. Third Attempt • 8 hidden units to 16 • 0.1 as learning rate (same) • 2500 iterations to 200 • Best on Test Set = 95.2% on 176th iteration • Meanwhile 98.7% on Training Set • 95.2% accuracy on 200th iteration • Best on Training Set = 98.7%on 176th iteration • Meanwhile 95.2% on Test Set

38. Graph of Third Attempt

39. Fourth Attempt • 16 hidden units (same) • 0.1 as learning rate to 0.05 • 200 iterations to 400 • Best on Test Set = 95.0% on 122th iteration • Meanwhile 98.4% on Training Set • 94.4% accuracy on 400th iteration • Best on Training Set = 98.7%on 394th iteration • Meanwhile 94.4% on Test Set

40. Graph of Fourth Attempt

41. Last Attempt • 16 hidden units to 32 • 0.05 as learning rate (same) • 400 iterations to 1000(4 hours and 38 minutes) • Best on Test Set = 96.4% on 315th iteration • Meanwhile 99.5% on Training Set • 96.0% accuracy on 1000th iteration • Best on Training Set = 99.9%on 945th iteration • Meanwhile 96.1% on Test Set

42. Graph of Last Attempt

43. Advantages • Handle errors well. • Graceful degradation. • Can learn novel solutions.

44. Disadvantages • Can’t understand how the learned network works, therefore they are the second best way to do something. • Need as many examples as possible. • Learning takes too much time / processing. • Sometimes the network may not converge.