Artificial Neural Networks - Basics

1. Business SchoolInstitute of Business Informatics ArtificialNeural Networks - Basics Uwe Lämmel www.wi.hs-wismar.de/~laemmel U.laemmel@wi.hs-wismar.de

2. Neural Networks • Idea • Artificial Neuron & Network • Supervised Learning • Unsupervised Learning • Data Mining – other Techniques

3. Idea An artificial Neuron Neural Network Example Learning Application Artificial Neuron & Network

4. Idea • A human being learns by example “learning by doing” • seeing(Perception), Walking, Speaking,… • Can a machine do the same? • A human being uses his brain. A brain consists of millions of single cells.A cell is connected with ten thousands of other cell. • Is it possible to simulate a similar structure on a computer?

5. Idea Artificial Neural Network • Information processing similar to processes in a mammal brain • heavy parallel systems, • able to learn • great number of simple cells ? Is it useful to copy nature ? • wheel, aeroplane, ...

6. Idea An artificial neural network functions in a similar way a natural neural network does. • we need: • software neurons • software connections between neurons • software learning algorithms

7. Dendrits cell and nucleus Axon (Neurit) Synapsis A biological Neuron • Dendrits: (Input) Getting other activations • Axon: (Output ) forward the activation (from 1mm up to 1m long) • Synapse: transfer of activation: • to other cells, e.g. Dendrits of other neurons • a cell has about 1.000 to 10.000 connections to other cells • Cell Nucleus: (processing) evaluation of activation

8. w1i w2i ... wji Dendrits cell and nucleus Axon oi (Neurit) Synapsis Natural vs. Artificial Neuron

9. Abstraction • Dendrits: weighted (real number) connections • Axon: output: real number • Synapse: --- (identity: output is directly forwarded) • Cell nucleus: unit contains simple functions input = (many) real numbers processing = activation function output = real number (~activation)

10. w1i w2i ... wji oi An artificial Neuron net : input from the network w : weight of a connection act : activation fact : activation function  :bias/threshold fout : output function (mostly ID) o : output

11. Exercise: AND/OR -LTU • Built a „network“ that works like an AND-function • Built an OR-network • Try to built an XOR-network LTU – Linear Threshold Unit

12. a1=__ a2=__ w1=__ w2=__ net = o1w1+o2 w2 a = 1, if net > = 0, otherwise o = a o=__ A simple switch • Neuron = AND-function • Find parameters: • Input neurons 1,2 :a1,a2 input pattern, • weights of edges: w1, w2 • bias  • Now evaluate output o !

13. Mathematics in a Cell • Propagation function (neuron input)neti(t) =  ojwj = w1i  o1 + w2i  o2 + ... • Activationai(t) – Activation at time t • Activation functionfact :ai(t+1) = fact(ai(t), neti(t), i) i – bias • Output functionfout :oi = fout(ai)

14. Activation

15. Bias function 1,0 0,5 0,0 -4,0 -2,0 0,0 2,0 4,0 -0,5 Identity -1,0 4,0 2,0 0,0 -4,0 -2,5 -1,0 0,5 2,0 3,5 -2,0 -4,0 Activation Functions activation functionsaresigmoid functions

16. y = tanh(c·x) c=3 1,0 c=2 c=1 0,5 y = 1/(1+exp(-c·x)) -0,6 0,6 1,0 -1,0 c=10 1,0 -0,5 c=3 c=1 -1,0 0,5 1,0 0,0 -1,0 Activation Functions activation functionsaresigmoid functions Logistic function:

17. Structure of a network • layers • input layer – input neurons • output layer – output neurons • hidden layer – hidden neurons • An n-layer network has: • n layer of connections which can be trained 

18. Definition: A Neural Network … • … is characterized by • connections of many (a lot of) simple units (neurons) and • units exchanging signals via these connections • … is a • coherent, directed graph which has • weighted edges and • each node (neuron, unit ) contains a value (activation).

19. XOR-network input output • weights are set by hand • fact = 1, net >  = 0, otherwise •  = 1.0

20. XOR-Example • standard propagation function:neti(t) =  oj(t)wji • activation function = bias function • ai = 1, if neti(t)> i e.g. =0.5 0, otherwise • output function = Identity: oj = aj • use EXCEL and built the XOR-example network!

21. Network evaluation network error learning examples changingnetwork parameters Learning

22. Learning - can be done by: • Modification of the weight of a connection • most frequently used • Deleting connections • can be done by (1): w=0  w0 • Modification of the bias of a neuron • Can be done by (1) using an extra neuron • changing functions (activation, propagation, output function) • Building new cells (GNG) • Building new connections • Deleting cells

23. Learning • supervised learning • We know the results for certain input pattern:teaching input  teaching output • Network error is used to adapt weights • Fast, but not natural • reinforced learning • We know whether output is right or wrong • Information is used to adapt weights • Slower than supervised; natural • unsupervised learning • Network has to learn by itself; • Slow , natural

24. Applications • Pattern recognition (text, numbers, faces) • Checking the quality of a surface • Control of autonomous vehicles • Monitoring of credit card accounts • Data Mining

25. Applications • Speech recognition • Control of artificial limbs • classification of galaxies • Product orders (Supermarket) • Forecast of energy consumption • Stock value forecast

26. Application - Summary • Classification • Clustering • Forecast • Pattern recognition • Learning by examples, generalization • Recognition of not known structures