Dimensions of Neural Networks

1. Dimensions of Neural Networks Ali Akbar Darabi Ghassem Mirroshandel Hootan Nokhost

2. Outline • Motivation • Neural Networks Power • Kolmogorov Theory • Cascade Correlation

3. Motivation • Consider you are an engineer and you know ANN • You encounter a problem that can not be solved with common analytical approaches • You decide to use ANN

4. But… • Some questions • Is this problem solvable using ANN? • How many neurons? • How many layers? • …

5. Two Approaches • Fundamental Analyses • Kolmogrov Theory • Adaptive Networks • Cascade Correlation

6. Outline • Motivation • Neural Networks Power • Kolmogorov Theory • Cascade Correlation

7. Single layer Networks • Limitations of the perceptron and linear classifiers

8. A Solution

9. Network Construction (x,y)→(x^2, y^2,x*y) 1 2

10. Network Construction (con…)

11. Network Construction (Con…)

12. Learning Mechanism • Using Error Function • Gradient Descent

13. Outline • Motivation • Neural Networks Power • Kolmogorov Theory • Cascade Correlation

14. Kolmogorov theorem (concept) • An example: • Any continuous function of n dimensions can be completely characterized by a dimensional continuous functions

15. g r y x An Idea • Suppose we want to construct f (x, y) • A simple idea: find a mapping • (x, y) → r • Then define a function g such that: • g(r) = f(x, y)

16. An Example • Suppose we have a discrete function: • We choose a mapping • We define the 1-dimentional function • So

17. Kolmogrov theorem • In the illustrated example we had:

18. Applying to the neural networks

19. Universal Approximation • Neural Networks with a hidden layer can approximate any continuous function with arbitrary precision • Use independent function from main function • approximate the network with traditional networks

20. A kolmogorov Network • We have to define: • Mapping • Function g

21. Spline Function • Linear combination of several 3-dimensional functions • Used to approximate functions with given points

22. Mapping y x

23. X2=4.5 X1=2.5 Example 2.1 1.6 x1 2.5 3.2 2.5 1.4 x2 4.5

24. X2=4.5 X1=2.5 Function g • Now for each unique input value of a we should define a output value g corresponding to f • We choose the value of f in the center of the square

25. Function g (Con…)

26. Reduce Error • Shifting defined patterns • N different patterns will be generated • Use avg y

27. Replace the function • With sufficiently large number of knots:

28. Outline • Motivation • Neural Networks Power • Kolmogorov Theory • Cascade Correlation

29. Cascade Correlation • Dynamic size, depth, topology • Single layer learning in spite of multilayer structure • Fast learning

30. Architecture

31. Algorithm step 1

32. Adding Hidden Layer

33. Correlation • Residual error for output unit for pattern p • Average residual error for output unit • Computed activation for input vector x(p) • Z(p) • Average activation, over all patterns, of candidate unit

34. Correlation • Use Variance as a similarity criteria • Update weights similar to gradient descent

35. Algorithm step 2

36. Adding Hiding Neuron

37. Algorithm Step 3

38. Final Result

39. An Example • 100 Run • 1700 epochs on avg • Beats standard backprob with factor 10 with the same complexity

40. Results • Cascade Correlation is either better • Only forward pass • Many of epochs are run while the network is very small • Cashing mechanism

41. Network Steps

42. Network Steps (con..)

43. Another Example • N-input parity problem • Standard backprob takes 2000 epoches on N=8 with 16 hidden neurons

44. Discussion • There is no need to guess the size, depth and the connectivity pattern • Learns fast • Can build deep networks (high order feature detector) • Herd effect • Results can be cashed

45. Conclusion • A Network with a hidden layer can define complex boundaries and can approximate any function • The number of neurons in the hidden layer determines the amount of approximation • Dynamic Networks