Neural networks for data mining

1. Neural networksfor data mining Eric Postma MICC-IKAT Universiteit Maastricht

2. Overview Introduction: The biology of neural networks • the biological computer • brain-inspired models • basic notions Interactive neural-network demonstrations • Perceptron • Multilayer perceptron • Kohonen’s self-organising feature map • Examples of applications

3. A typical AI agent

4. Two types of learning • Supervised learning • curve fitting, surface fitting, ... • Unsupervised learning • clustering, visualisation...

5. An input-output function

6. Fitting a surface to four points

7. Regression

8. Classification

9. The history of neural networks • A powerful metaphor • Several decades of theoretical analyses led to the formalisation in terms of statistics • Bayesian framework • We discuss neural networks from the original metaphorical perspective

10. (Artificial) neural networks The digital computer versus the neural computer

11. The Von Neumann architecture

12. The biological architecture

13. Digital versus biological computers 5 distinguishing properties • speed • robustness • flexibility • adaptivity • context-sensitivity

14. Speed: The “hundred time steps” argument The critical resource that is most obvious is time. Neurons whose basic computational speed is a few milliseconds must be made to account for complex behaviors which are carried out in a few hudred milliseconds (Posner, 1978). This means that entire complex behaviors are carried out in less than a hundred time steps. Feldman and Ballard (1982)

15. Graceful Degradation performance damage

16. Flexibility: the Necker cube

17. vision = constraint satisfaction

18. And sometimes plain search…

19. Adaptivitiy processing implies learning in biological computers versus processing does not imply learning in digital computers

20. Context-sensitivity: patterns emergent properties

21. Robustness and context-sensitivitycoping with noise

22. The neural computer • Is it possible to develop a model after the natural example? • Brain-inspired models: • models based on a restricted set of structural en functional properties of the (human) brain

23. The Neural Computer (structure)

24. Neurons, the building blocks of the brain

26. Neural activity out in

27. Synapses,the basis of learning and memory

28. Learning: Hebb’s rule neuron 1 synapse neuron 2

29. Forgetting in neural networks

30. Towards neural networks

31. Connectivity An example: The visual system is a feedforward hierarchy of neural modules Every module is (to a certain extent) responsible for a certain function

32. (Artificial) Neural Networks • Neurons • activity • nonlinear input-output function • Connections • weight • Learning • supervised • unsupervised

33. Artificial Neurons • input (vectors) • summation (excitation) • output (activation) i

34. 1 f(x) = 1 + e -x/a Input-output function • nonlinear function: a  0 f(e) a   e

35. wAB A B Artificial Connections (Synapses) • wAB • The weight of the connection from neuron A to neuron B

36. The Perceptron

37. Learning in the Perceptron • Delta learning rule • the difference between the desired output tand the actual output o, given inputx • Global error E • is a function of the differences between the desired and actual outputs

38. Gradient Descent

39. Linear decision boundaries

40. Minsky and Papert’s connectedness argument

41. The history of the Perceptron • Rosenblatt (1959) • Minsky & Papert (1961) • Rumelhart & McClelland (1986)

42. The multilayer perceptron input output one or more hidden layers

43. Training the MLP • supervised learning • each training pattern: input + desired output • in each epoch: present all patterns • at each presentation: adapt weights • after many epochs convergence to a local minimum

44. phoneme recognition with a MLP Output: pronunciation input: frequencies

45. Non-linear decision boundaries

46. Compression with an MLPthe autoencoder

47. hidden representation

48. Restricted Boltzmann machines (RBMs)

49. Learning in the MLP