Implementing Local Relative Sensitivity Pruning

1. Implementing Local Relative Sensitivity Pruning Paul Victorey

2. Why Prune? • It is believed that neural networks with too many connections tend to have more capacity for learning training set error. • Pruning has been shown to reduce testing error. • Pruning has been shown to yield a more favorable generalization factor (closer to 1).

3. Pruning Strategies • Optimal Brain Surgeon (OBS) and Optimal Brain Damage (OBD) are widely used, but are computationally expensive. • Karnin proposes using “Sensitivity value” (SV) as pruning criteria. The SV is defined as: • Local Relative Sensitivity Pruning (LRSP) is similar to the SV approach. It considers local as well as global effects of pruning. The LRSI is the SV of the given weight divided by the sum of the SVs of the weight’s local group; a local group is the collection of weights which are associated with the inputs to a single neuron.

4. LRSP Algorithm • Partition data into training, validation, and testing sets. • Use training set to train a fully connected MLP. • Compute the LRSI for each weight. • Prune at most one input to any given node; prune only if a weight has a LRSI less than a threshold (0.01 seems to work). • Retrain the pruned network. Evaluate the performance of this network compared to the network pre-pruning. If the network is better, keep the new network, and perform pruning again. When the network shows no improvement, stop. • Several possible cost functions to determine network fitness exist. Most popular: Validation error, Generalization Factor (when determining GF, validation error is used to approximate testing error)

5. Results • Ponnapalli et al have shown that this method is preferable to the SV-pruning proposed by Karnin • Networks pruned by LRSP in many cases approach a unity GF, and have a better testing error than pre-pruning. • The positive results, coupled with a low computational overhead, make LRSP a good choice of pruning algorithm.