EE645 Neural Networks and Learning Theory

EE645Neural Networks and Learning Theory Spring 2003 Prof. Anthony Kuh Dept. of Elec. Eng. University of Hawaii Phone: (808)-956-7527, Fax: (808)-956-3427 Email: kuh@spectra.eng.hawaii.edu

I. Introduction to neural networks Goal: study computational capabilities of neural network and learning systems. Multidisciplinary field Algorithms, Analysis, Applications

A. Motivation Why study neural networks and machine learning? Biological inspiration (natural computation) Nonparametric models: adaptive learning systems, learning from examples, analysis of learning models Implementation Applications Cognitive (Human vs. Computer Intelligence): Humans superior to computers in pattern recognition, associative recall, learning complex tasks. Computers superior to humans in arithmetic computations, simple repeatable tasks. Biological: (study human brain) 10^10 to 10^11 neurons in cerebral cortex with on average of 10^3 interconnections / neuron.

A neuron Schematic of one neuron

Neural Network Connection of many neurons together forms a neural network. • Neural network properties: • Highly parallel (distributed computing) • Robust and fault tolerant • Flexible (short and long term learning) • Handles variety of information (often random, fuzzy, and inconsistent) • Small, compact, dissipates very little power

B. Single Neuron (Computational node) s=w T x + w0 ; synaptic strength (linearly weighted sum of inputs). y=g(s); activation or squashing function g( ) w  x y s w 0

Activation functions • Linear units: g(s) = s. • Linear threshold units: g(s) = sgn (s). • Sigmoidal units: g(s) = tanh (Bs), B >0. Neural networks generally have nonlinear activation functions. Most popular models: linear threshold units and sigmoidal units. Other types of computational units : receptive units (radial basis functions).

C. Neural Network Architectures Systems composed of interconnected neurons output inputs Neural network represented by directed graph: edges represent weights and nodes represent computational units.

Definitions • Feedforward neural network has no loops in directed graph. Neural networks are often arranged in layers. • Single layer feedforward neural network has one layer of computational nodes. • Multilayer feedforward neural network has two or more layers of computational nodes. • Computational nodes that are not output nodes are called hidden units.

D. Learning and Information Storage • Neural networks have computational capabilities. Where is information stored in a neural network? What are parameters of neural network? • How does a neural network work? (two phases) Training or learning phase (equivalent to write phase in conventional computer memory): weights are adjusted to meet certain desired criterion. Recall or test phase (equivalent to read phase in conventional computer memory): weights are fixed as neural network realizes some task.

Learning and Information (continued) 3) What can neural network models learn? Boolean functions Pattern recognition problems Function approximation Dynamical systems 4) What type of learning algorithms are there? Supervised learning (learning with a teacher) Unsupervised learning (no teacher) Reinforcement learning (learning with a critic)

Learning and Information (continued) 5) How do neural networks learn? • Iterative algorithm: weights of neural network are adjusted on-line as training data is received. w(k+1) =L(w(k),x(k),d(k)) for supervised learning where d(k) is desired output. • Need cost criterion: common cost criterion Mean Squared Error: for one output J(w) =  (y(k) – d(k)) 2 • Goal is to find minimum J(w) over all possible w. Iterative techniques often use gradient descent approaches.

Learning and Information (continued) 6)Learning and Generalization Learning algorithm takes training examples as inputs and produces concept, pattern or function to be learned. How good is learning algorithm? Generalization ability measures how well learning algorithm performs. Sufficient number of training examples. (LLN, typical sequences) Occam’s razor: “simplest explanation is the best”. + + + + + + Regression problem

Learning and Information (continued) Generalization error g = emp + model Empirical error: average error from training data (desired output vs. actual output) Model error: due to dimensionality of class of functions or patterns Desire class to be large enough so that empirical error is small and small enough so that model error is small.

II. Linear threshold units A. Preliminaries sgn( ) w  x y s w 0 1, if s>=0 -1, if s<0 sgn(s)=

Linearly separable Consider a set of points with two labels: +and o. Set of points is linearly separable if a linear threshold function can partition the + points from the o points. o + + o o Set of linearly separable points +

Not linearly separable A set of labeled points that cannot be partitioned by a linear threshold function is not linearly separable. o + + o Set of points that are not linearly separable

B. Perceptron Learning Algorithm An iterative learning algorithm that can find linear threshold function to partition two set of points. • w(0) arbitrary • Pick point (x(k),d(k)). • If w(k) T x(k)d(k) > 0 go to 5) • w(k+1) = w(k ) + x(k)d(k) • k=k+1, check if cycled through data, if not go to 2 • Otherwise stop.

PLA comments • Perceptron convergence theorem (requires margins) • Sketch of proof • Updating threshold weights • Algorithm is based on cost function J(w) = - (sum of synaptic strengths of misclassified points) w(k+1) = w(k) - (k)J(w(k)) (gradient descent)

Perceptron Convergence Theorem Assumptions: w* solutions and ||w*||=1, no threshold and w(0)=0. Let max||x(k)||= and min y(k)x(k)Tw*=. <w(k),w*>=<w(k-1) + x(k-1)y(k-1),w*>  <w(k-1),w*> +   k . ||w(k)||2  ||w(k-1)||2 +||x(k-1)||2 ||w(k-1)||2 +  2  k 2 . Implies that k  ( /  ) 2 (max number of updates).

III. Linear Units A. Preliminaries w  x s=y

Model Assumptions and Parameters • Training examples (x(k),d(k)) drawn randomly • Parameters • Inputs: x(k) • Outputs: y(k) • Desired outputs: d(k) • Weights: w(k) • Error: e(k)= d(k)-y(k) • Error criterion (MSE) min J(w) = E [.5(e(k)) 2]

Wiener solution Define P= E(x(k)d(k)) and R=E(x(k)x(k)T). J(w) =.5 E[(d(k)-y(k))2] = .5E(d(k)2)- E(x(k)d(k)) Tw +wT E(x(k)x(k) T)w = .5E[d(k) 2] –PTw +.5wTRw Note J(w) is a quadratic function of w. To minimize J(w) find gradient, J(w) and set to 0. J(w) = -P + Rw = 0 Rw=P (Wiener solution) If R is nonsingular, then w= R-1 P. Resulting MSE = .5E[d(k)2]-PTR-1P

Iterative algorithms • Steepest descent algorithm (move in direction of negative gradient) w(k+1) = w(k) - J(w(k)) = w(k) +  (P-Rw(k)) • Least mean square algorithm (approximate gradient from training example) J(w(k))= -e(k)x(k) w(k+1) = w(k) + e(k)x(k) ^

Steepest Descent Convergence • w(k+1) = w(k) +  (P-Rw(k)); Let w* be solution. Center weight vector v=w-w* • v(k+1) = v(k) -  (Rw(k)); Assume R is nonsingular. Decorrelate weight vector u= Q-1v where R=Q Q-1 is the transformation that diagonalizes R. • u(k+1) = (I -   ), u(k) = (I -   )k u(0). Conditions for convergence 0<  < 2/max .

LMS Algorithm Properties • Steepest Descent and LMS algorithm convergence depends on step size  and eigenvalues of R. • LMS algorithm is simple to implement. • LMS algorithm convergence is relatively slow. • Tradeoff between convergence speed and excess MSE. • LMS algorithm can track training data that is time varying.

Adaptive MMSE Methods • Training data • Linear MMSE: LMS, RLS algorithms • Nonlinear Decision feedback detectors • Blind algorithms • Second order statistics • Minimum Output Energy Methods • Reduced order approximations: PCA, multistage Wiener Filter • Higher order statistics • Cumulants, Information based criteria

Designing a learning system Given a set of training data, design a system that can realize the desired task. Signal Processing Feature Extraction Neural Network Outputs Inputs

IV. Multilayer Networks • A. Capabilities • Depend directly on total number of weights and threshold values. • A one hidden layer network with sufficient number of hidden units can arbitrarily approximate any boolean function, pattern recognition problems, and well behaved function approximation problems. • Sigmoidal units more powerful than linear threshold units.

B. Error backpropagation Error backpropagation algorithm: methodical way of implementing LMS algorithm for multilayer neural networks. • Two passes: forward pass (computational pass), backward pass (weight correction pass). • Analog computations based on MSE criterion. • Hidden units usually sigmoidal units. • Initialization: weights take on small random values. • Algorithm may not converge to global minimum. • Algorithm converges slower than for linear networks. • Representation is distributed.

BP Algorithm Comments • s are error terms computed from output layer back to first layer in dual network. • Training is usually done online. • Examples presented in random or sequential order. • Update rule is local as weight changes only involve connections to weight. • Computational complexity depends on number of computational units. • Initial weights randomized to avoid converging to local minima.

BP Algorithm Comment continued • Threshold weights updated in similar manner to other weights (input =1). • Momentum term added to speed up convergence. • Step size set to small value. • Sigmoidal activation derivatives simple to compute.

BP Architecture Output of computational values calculated Forward network Output of error terms calculated Sensitivity network

Modifications to BP Algorithm • Batch procedure • Variable step size • Better approximation of gradient method (momentum term, conjugate gradient) • Newton methods (Hessian) • Alternate cost functions • Regularization • Network construction algorithms • Incorporating time

When to stop training First major features captured. As training continues minor features captured. Look at training error. Crossvalidation (training, validation, and test sets) testing error training error Learning typically slow and may find flat learning areas with little improvement in energy function.

C. Radial Basis Functions • Use locally receptive units (potential functions) • Transform input space to hidden unit space via potential functions. • Output unit is linear.   output inputs Linear unit  Potential units (x) = exp (-.5||x-c|| 2 / 2

Transformation of input space  X X O X O O X O Input space Feature space : X Z

Training Radial basis functions • Use gradient descent on unknown parameters: centers, widths, and output weights • Separate tasks for quicker training: (first layer centers, widths), (second layer weights) • First layer • Fix widths, centers determined from lattice structure • Fix widths, clustering algorithm for centers • Resource allocation network • Second layer: use LMS to learn weights

Comparisons between RBFs and BP Algorithm • RBF single hidden layer and BP algorithm can have many hidden layers. • RBF (potential functions) locally receptive units versus BP algorithm (sigmoidal units) distributed representations. • RBF typically many more hidden units. • RBF training typically quicker training.

V. Alternate Detection Method Consider detection methods based on optimum margin classifiers or Support Vector Machines (SVM) • SVM are based on concepts from statistical learning theory. • SVM are easily extended to nonlinear decision regions via kernel functions. • SVM solutions involve solving quadratic programming problems.

Optimal Marginal Classifiers X Given a set of points that are linearly separable: X X X Which hyperplane should you choose to separate points? O O O Choose hyperplane that maximizes distance between two sets of points.

Finding Optimal Hyperplane margins • Draw convex hull around each set of points. • Find shortest line segment connecting two convex hulls. • Find midpoint of line segment. • Optimal hyperplane intersects line segment at midpoing perpendicular to line segment. X X X w X O O O Optimal hyperplane

Alternative Characterization of Optimal Margin Classifiers Maximizing margins equivalent to minimizing magnitude of weight vector. 2m X X X T W (u-v) = 2 w T X W (u-v)/ W = 2/ W =2m u T O W u+ b = 1 O v O T W v+ b = -1

Solution in 1 Dimension O O O O O X O X X O X X X Points on wrong side of hyperplane If C is large SV include If C is small SV include all points (scaled MMSE solution) Note that weight vector depends most heavily on outer support vectors.

Comments on 1 Dimensional Solution • Simple algorithm can be implemented to solve 1D problem. • Solution in multiple dimensions is finding weight and then projecting down to 1D. • Min. probability of error threshold depends on likelihood ratio. • MMSE solution depends on all points where as SVM depends on SV (points that are under margin (closer to min. probability of error). • Min. probability of error, MMSE solution, and SVM in general give different detectors.

Kernel Methods In many classification and detection problems a linear classifier is not sufficient. However, working in higher dimensions can lead to “curse of dimensionality”. Solution: Use kernel methods where computations done in dual observation space.  X X O X O O X O Input space Feature space : X Z

Solving QP problem SVM require solving large QP problems. However, many s are zero (not support vectors). Breakup QP into subproblem. • Chunking : (Vapnik 1979) numerical solution. • Ossuna algorithm: (1997) numerical solution. • Platt algorithm: (1998) Sequential Minimization Optimization (SMO) analytical solution.

SMO Algorithm Sequential Minimization Optimization breaks up QP program into small subproblems that are solved analytically. • SMO solves dual QP SVM problem by examining points that violate KKT conditions. • Algorithm converges and consists of: • Search for 2 points that violate KKT conditions. • Solve QP program for 2 points. • Calculate threshold value b. Continue until all points satisfy KKT conditions. • On numerous benchmarks time to convergence of SMO varied from O (l) to O (l 2.2 ) . Convergence time depends on difficulty of classification problem and kernel functions used.

SVM Summary • SVM are based on optimum margin classifiers and are solved using quadratic programming methods. • SVM are easily extended to problems that are not linearly separable. • SVM can create nonlinear separating surfaces via kernel functions. • SVM can be efficiently programmed via the SMO algorithm. • SVM can be extended to solve regression problems.

VI.Unsupervised Learning • Motivation • Given a set of training examples with no teacher or • critic, why do we learn? • Feature extraction • Data compression • Signal detection and recovery • Self organization • Information can be found about data from inputs.

EE645 Neural Networks and Learning Theory