Neural Networks

1. Neural Networks Pabitra Mitra Computer Science and Engineering IIT Kharagpur pabitra@gmail.com

3. The Neuron • The neuron is the basic information processing unit of a NN. It consists of: • A set of synapses or connecting links, each link characterized by a weight: W1, W2, …, Wm • An adder function (linear combiner) which computes the weighted sum of the inputs: • Activation function (squashing function) for limiting the amplitude of the output of the neuron. NN 1

4. Computation at Units • Compute a 0-1 or a graded function of the weighted sum of the inputs • is the activation function

5. Bias b x1 w1 Activation function Local Field v Output y Input signal x2 w2 Summing function xm wm Synaptic weights The Neuron NN 1

6. Common Activation Functions • Step function: g(x)=1, if x >= t ( t is a threshold) g(x) = 0, if x < t • Sign function: g(x)=1, if x >= t ( t is a threshold) g(x) = -1, if x < t • Sigmoid function: g(x)= 1/(1+exp(-x))

7. v u Bias of a Neuron • Bias b has the effect of applying an affine transformation to u v = u + b • vis the induced field of the neuron NN 1

8. Bias as extra input w0 x0 = +1 x1 w1 Activation function Local Field v Input signal Output y x2 w2 Summing function Synaptic weights xm wm • Bias is an external parameter of the neuron. Can be modeled by adding an extra input. NN 1

9. Face Recognition 90% accurate learning head pose, and recognizing 1-of-20 faces NN 1

10. Handwritten digit recognition NN 1

11. error: +1 = cat, -1 = dog y cat dog y x2 x1 x2 x1 perceptual features Computing with spaces

12. Can Implement Boolean Functions • A unit can implement And, Or, and Not • Need mapping True and False to numbers: • e.g. True = 1.0, False= 0.0 • (Exercise) Use a step function and show how to implement various simple Boolean functions • Combining the units, we can get any Boolean function of n variables Can obtain logical circuits as special case

13. Network Structures • Feedforward (no cycles), less power, easier understood • Input units • Hidden layers • Output units • Perceptron: No hidden layer, so basically correspond to one unit, also basically linear threshold functions (ltf) • Ltf: defined by weights and threshold , value is 1 iff otherwise, 0

14. Single Layer Feed-forward Input layer of source nodes Output layer of neurons NN 1

15. Multi layer feed-forward 3-4-2 Network Output layer Input layer Hidden Layer NN 1

16. Network Structures • Recurrent (cycles exist), more powerful as they can implement state, but harder to analyze. Examples: • Hopfield network, symmetric connections, interesting properties, useful for implementing associative memory • Boltzmann machines: more general, with applications in constraint satisfaction and combinatorial optimization

17. output layer copy context units input input layer Simple recurrent networks x1 x2 hidden layer z1 z2 x1 x2 (Elman, 1990)

18. Perceptron Capabilities • Quite expressive: many, but not all Boolean functions can be expressed. Examples: • conjuncts and disjunctions, example • more generally, can represent functions that are true if and only if at least k of the inputs are true: • Can’t represent XOR

19. Representable Functions • Perceptrons have a monotinicity property: If a link has positive weight, activation can only increase as the corresponding input value increases (irrespective of other input values) • Can’t represent functions where input interactions can cancel one another’s effect (e.g. XOR)

20. Representable Functions • Can represent only linearly separable functions • Geometrically: only if there is a line (plane) separating the positives from the negatives • The good news: such functions are PAC learnable and learning algorithms exist

21. Linearly Separable - + + + _ + + + + + + + + +

22. NOT linearly Separable + + + _ + + OR + + +

23. OR AND x2 x2 x1 x1 XOR x2 x1 Problems with simple networks y x2 x1 x1 x2 Some kinds of data are not linearly separable

24. output layer y z2 z1 x2 input layer x1 A solution: multiple layers y hidden layer z1 z2 x1 x2

25. The Perceptron Learning Algorithm • Example of current-best-hypothesis (CBH) search (so incremental, etc.): • Begin with a hypothesis (a perceptron) • Repeat over all examples several times • Adjust weights as examples are seen • Until all examples correctly classified or a stopping criterion reached

26. Method for Adjusting Weights • One weight update possibility: • If classification correct, don’t change • Otherwise: • If false negative, add input: • If false positive, subtract input: • Intuition: For instance, if example is positive, strengthen/increase the weights corresponding to the positive attributes of the example

27. Properties of the Algorithm • In general, also apply a learning rate • The adjustment is in the direction of minimizing error on the example • If learning rate is appropriate and the examples are linear separable, after a finite number of iterations, the algorithm converges to a linear separator

28. Another Algorithm(least-sum-squares algorithm) • Define and minimize an error function • S is the set of examples, is the ideal function, is the linear function corresponding to the current perceptron • Error of the perceptron (over all examples): • Note:

29. influence of input output error The Delta Rule +1 = cat, -1 = dog y for any function g with derivativeg x1 x2 perceptual features

30. Derivative of Error • Gradient (derivative) of E: • Take the steepest descent direction: • is the gradient along , is the learning rate

31. Gradient Descent • The algorithm: pick initial random perceptron and repeatedly compute error and modify the perceptron (take a step along the reverse of gradient) E Gradient direction: Descent direction:

32. General-purpose learning mechanisms E (error) ( is learning rate) wij

33. Gradient Calculation

34. Derivation (cont.)

35. Properties of the algorithm • Error function has no local minima (is quadratic) • The algorithm is a gradient descent method to the global minimum, and will asymptotically converge • Even if not linearly separable, can find a good (minimum error) linear classifier • Incremental?

36. Multilayer Feed-Forward Networks • Multiple perceptrons, layered • Example: a two-layer network with 3 inputs one output, one hidden layer (two hidden units) output layer inputs layer hidden layer

37. Power/Expressiveness • Can represent interactions among inputs (unlike perceptrons) • Two layer networks can represent any Boolean function, and continuous functions (within a tolerance) as long as the number of hidden units is sufficient and appropriate activation functions used • Learning algorithms exist, but weaker guarantees than perceptron learning algorithms

38. Back-Propagation • Similar to the perceptron learning algorithm and gradient descent for perceptrons • Problem to overcome: How to adjust internal links (how to distribute the “blame” or the error) • Assumption: internal units use differentiable functions and nonlinear • sigmoid functions are convenient

39. z-1 z-1 z-1 Recurrent network Recurrent Network with hidden neuron(s): unit delay operator z-1 implies dynamic system input hidden output NN 1

40. Back-Propagation (cont.) • Start with a network with random weights • Repeat until a stopping criterion is met • For each example, compute the network output and for each unit i it’s error term • Update each weight (weight of link going from node i to node j): Output of unit i

41. The Error Term

42. Derivation • Write the error for a single training example; as before use sum of squared error (as it’s convenient for differentiation, etc): • Differentiate (with respect to each weight…) • For example, we get for weight connecting node j to output i

43. Properties • Converges to a minimum, but could be a local minimum • Could be slow to converge (Note: Training a three node net is NP-Complete!) • Must watch for over-fitting just as in decision trees (use validation sets, etc.) • Network structure? Often two layers suffices, start with relatively few hidden units

44. Properties (cont.) • Many variations to the basic back-propagation: e.g. use momentum • Reduce with time (applies to perceptrons as well) Nth update amount a constant

45. Networks, features, and spaces • Artificial neural networks can represent any continuous function… • Simple algorithms for learning from data • fuzzy boundaries • effects of typicality

46. NN properties • Can handle domains with • continuous and discrete attributes • Many attributes • noisy data • Could be slow at training but fast at evaluation time • Human understanding of what the network does could be limited

47. Networks, features, and spaces • Artificial neural networks can represent any continuous function… • Simple algorithms for learning from data • fuzzy boundaries • effects of typicality • A way to explain how people could learn things that look like rules and symbols…

48. Networks, features, and spaces • Artificial neural networks can represent any continuous function… • Simple algorithms for learning from data • fuzzy boundaries • effects of typicality • A way to explain how people could learn things that look like rules and symbols… • Big question: how much of cognition can be explained by the input data?

49. Challenges for neural networks • Being able to learn anything can make it harder to learn specific things • this is the “bias-variance tradeoff”