Back-propagation

1. Back-propagation Chih-yun Lin 6/6/2014

2. Agenda • Perceptron vs. back-propagation network • Network structure • Learning rule • Why a hidden layer? • An example: Jets or Sharks • Conclusions

3. Network Structure –Perceptron O Output Unit Wj IjInput Units

4. Network Structure –Back-propagation Network Oi Output Unit Wj,i ajHidden Units Wk,j Ik Input Units

5. Learning Rule • Measure error • Reduce that error • By appropriately adjusting each of the weights in the network

6. Learning Rule –Perceptron • Err = T–O • O is the predicted output • T is the correct output • WjWj+ α * Ij* Err • Ij is the activation of a unit j in the input layer • α is a constant called the learning rate

7. Learning Rule –Back-propagation Network • Erri = Ti–Oi • Wj,iWj,i+ α * aj * Δi • Δi = Erri * g’(ini) • g’ is the derivative of the activation function g • ajis the activation of the hidden unit • Wk,jWk,j+ α * Ik * Δj • Δj = g’(inj) * ΣiWj,i* Δi

8. Learning Rule –Back-propagation Network • E = 1/2Σi(Ti–Oi)2 • = - Ik * Δj

9. Why a hidden layer? • (1 w1) + (1 w2) < ==> w1 + w2 < • (1 w1) + (0 w2) > ==> w1 > • (0 w1) + (1 w2) > ==> w2 > • (0 w1) + (0 w2) < ==> 0 <

10. Why a hidden layer? (cont.) • (1 w1) + (1 w2) + (1 w3) < ==> w1 + w2 + w3 < • (1 w1) + (0 w2) + (0 w3) > ==> w1 > • (0 w1) + (1 w2) + (0 w3) > ==> w2 > • (0 w1) + (0 w2) + (0 w3) < ==> 0 <

11. An example: Jets or Sharks

12. Conclusion • Expressiveness: • Well-suited for continuous inputs,unlike most decision tree systems • Computational efficiency: • Time to error convergence is highly variable • Generalization: • Have reasonable success in a number of real-world problems

13. Conclusions (cont.) • Sensitivity to noise: • Very tolerant of noise in the input data • Transparency: • Neural networks are essentially black boxes • Prior knowledge: • Hard to used one’s knowledge to “prime” a network to learn better