230 likes | 359 Views
Forward & Backward selection in hybrid network. Introduction. A training algorithm for an hybrid neural network for regression. Hybrid neural network has hidden layer that has RBF or projection units (Perceptrons). When is it good?. Hidden Units. RBF:. MLP:. Overall algorithm.
E N D
Introduction • A training algorithm for an hybrid neural network for regression. • Hybrid neural network has hidden layer that has RBF or projection units (Perceptrons).
Hidden Units • RBF: • MLP:
Overall algorithm • Divide input space and assign units to each sub-region. • Optimize parameters. • Prune un-necessary weights using Bayesian Information Criteria.
Forward leg • Divide the input space into sub-regions • Select type of hidden unit for each sub-region • Stop when error goal or maximum number of units is achieved.
Input space division • Like CART using • Maximum reduction in
Units parameters • RBF unit: center at maximum point. • Projection unit: weight normalized of maximum point
Pruning • Target function values corrupted with Gaussian noise
BIC approximation • Schwartz, Kass and Raftery
Evidence Unit Type alg4. • Initialize alfa and beta • Loop: compute w,wo • Recompute alfa and beta • Until difference in the evidence is low.
Pumadyn data set DELVE archive • Dynamic of a puma robot arm. • Target: annular acceleration of one of the links. • Inputs: various joint angles, velocities and torques. • Large Guassian noise. • Data set non linear. • Input dimension: 8, 32.
Related work • Hassibi et al. with Optimal Brain Surgeon • Mackey with Bayesian inference of weights and regularization parameters. • HME Jordan and Jacob, division on input space. • Kass & Raftery Schwarz with BIC.
Discussion • Pruning removes 90% of parameters. • Pruning reduces variance of estimator. • The pruning algorithm is slow. • PRBFN better then MLP of RBF alone. • Bayesian techniques disadvantage: the prior distribution parameter. • Bayesian techniques are better then LRT. • Unit type selection is a crucial element in PRBFN • Curse of dimensionality is well seen on pumadyn data sets.