Understanding Bias-Variance Trade-Off in Machine Learning

1. CS 2750: Machine LearningLine Fitting + Bias-Variance Trade-off Prof. Adriana KovashkaUniversity of Pittsburgh January 26, 2017

2. Generalization • How well does a learned model generalize from the data it was trained on to a new test set? Training set (labels known) Test set (labels unknown) Slide credit: L. Lazebnik

3. Generalization • Components of expected loss • Noise in our observations: unavoidable • Bias: how much the average model over all training sets differs from the true model • Error due to inaccurate assumptions/simplifications made by the model • Variance: how much models estimated from different training sets differ from each other • Underfitting: model is too “simple” to represent all the relevant class characteristics • High bias and low variance • High training error and high test error • Overfitting: model is too “complex” and fits irrelevant characteristics (noise) in the data • Low bias and high variance • Low training error and high test error Adapted from L. Lazebnik

4. Bias-Variance Trade-off • Models with too few parameters are inaccurate because of a large bias (not enough flexibility). • Models with too many parameters are inaccurate because of a large variance (too much sensitivity to the sample). Purple dots = possible test points Red dots = training data (all that we see before we ship off our model!) Green curve = true underlying model Blue curve = our predicted model/fit Adapted from D. Hoiem

5. Polynomial Curve Fitting Slide credit: Chris Bishop

6. Sum-of-Squares Error Function Slide credit: Chris Bishop

7. 0th Order Polynomial Slide credit: Chris Bishop

8. 1st Order Polynomial Slide credit: Chris Bishop

9. 3rd Order Polynomial Slide credit: Chris Bishop

10. 9th Order Polynomial Slide credit: Chris Bishop

11. Over-fitting Root-Mean-Square (RMS) Error: Slide credit: Chris Bishop

12. Data Set Size: 9th Order Polynomial Slide credit: Chris Bishop

13. Data Set Size: 9th Order Polynomial Slide credit: Chris Bishop

14. Regularization Penalize large coefficient values (Remember: We want to minimize this expression.) Adapted from Chris Bishop

15. Regularization: Slide credit: Chris Bishop

16. Regularization: Slide credit: Chris Bishop

17. Polynomial Coefficients Slide credit: Chris Bishop

18. Polynomial Coefficients No regularization Huge regularization Adapted from Chris Bishop

19. Regularization: vs. Slide credit: Chris Bishop

20. Training vs test error Underfitting Overfitting Error Test error Training error High Bias Low Variance Complexity Low Bias High Variance Slide credit: D. Hoiem

21. The effect of training set size Few training examples Test Error Many training examples High Bias Low Variance Complexity Low Bias High Variance Slide credit: D. Hoiem

22. The effect of training set size Fixed prediction model Error Testing Generalization Error Training Number of Training Examples Adapted from D. Hoiem

23. Choosing the trade-off between bias and variance • Need validation set (separate from the test set) Validation error Error Training error High Bias Low Variance Complexity Low Bias High Variance Slide credit: D. Hoiem

24. Bias-variance (Bishop Sec. 3.2) Figure from Chris Bishop

25. How to reduce variance? • Get more training data • Regularize the parameters • Choose a simpler classifier Slide credit: D. Hoiem

26. Remember… • Three kinds of error • Inherent: unavoidable • Bias: due to over-simplifications • Variance: due to inability to perfectly estimate parameters from limited data • Try simple classifiers first • Use increasingly powerful classifiers with more training data (bias-variance trade-off) Adapted from D. Hoiem