1 / 26

Understanding Bias-Variance Trade-Off in Machine Learning

Learn about the bias-variance trade-off in machine learning and its impact on model generalization. Explore the concepts of underfitting and overfitting, and discover strategies to reduce variance.

caitlinj
Download Presentation

Understanding Bias-Variance Trade-Off in Machine Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CS 2750: Machine LearningLine Fitting + Bias-Variance Trade-off Prof. Adriana KovashkaUniversity of Pittsburgh January 26, 2017

  2. Generalization • How well does a learned model generalize from the data it was trained on to a new test set? Training set (labels known) Test set (labels unknown) Slide credit: L. Lazebnik

  3. Generalization • Components of expected loss • Noise in our observations: unavoidable • Bias: how much the average model over all training sets differs from the true model • Error due to inaccurate assumptions/simplifications made by the model • Variance: how much models estimated from different training sets differ from each other • Underfitting: model is too “simple” to represent all the relevant class characteristics • High bias and low variance • High training error and high test error • Overfitting: model is too “complex” and fits irrelevant characteristics (noise) in the data • Low bias and high variance • Low training error and high test error Adapted from L. Lazebnik

  4. Bias-Variance Trade-off • Models with too few parameters are inaccurate because of a large bias (not enough flexibility). • Models with too many parameters are inaccurate because of a large variance (too much sensitivity to the sample). Purple dots = possible test points Red dots = training data (all that we see before we ship off our model!) Green curve = true underlying model Blue curve = our predicted model/fit Adapted from D. Hoiem

  5. Polynomial Curve Fitting Slide credit: Chris Bishop

  6. Sum-of-Squares Error Function Slide credit: Chris Bishop

  7. 0th Order Polynomial Slide credit: Chris Bishop

  8. 1st Order Polynomial Slide credit: Chris Bishop

  9. 3rd Order Polynomial Slide credit: Chris Bishop

  10. 9th Order Polynomial Slide credit: Chris Bishop

  11. Over-fitting Root-Mean-Square (RMS) Error: Slide credit: Chris Bishop

  12. Data Set Size: 9th Order Polynomial Slide credit: Chris Bishop

  13. Data Set Size: 9th Order Polynomial Slide credit: Chris Bishop

  14. Regularization Penalize large coefficient values (Remember: We want to minimize this expression.) Adapted from Chris Bishop

  15. Regularization: Slide credit: Chris Bishop

  16. Regularization: Slide credit: Chris Bishop

  17. Polynomial Coefficients Slide credit: Chris Bishop

  18. Polynomial Coefficients No regularization Huge regularization Adapted from Chris Bishop

  19. Regularization: vs. Slide credit: Chris Bishop

  20. Training vs test error Underfitting Overfitting Error Test error Training error High Bias Low Variance Complexity Low Bias High Variance Slide credit: D. Hoiem

  21. The effect of training set size Few training examples Test Error Many training examples High Bias Low Variance Complexity Low Bias High Variance Slide credit: D. Hoiem

  22. The effect of training set size Fixed prediction model Error Testing Generalization Error Training Number of Training Examples Adapted from D. Hoiem

  23. Choosing the trade-off between bias and variance • Need validation set (separate from the test set) Validation error Error Training error High Bias Low Variance Complexity Low Bias High Variance Slide credit: D. Hoiem

  24. Bias-variance (Bishop Sec. 3.2) Figure from Chris Bishop

  25. How to reduce variance? • Get more training data • Regularize the parameters • Choose a simpler classifier Slide credit: D. Hoiem

  26. Remember… • Three kinds of error • Inherent: unavoidable • Bias: due to over-simplifications • Variance: due to inability to perfectly estimate parameters from limited data • Try simple classifiers first • Use increasingly powerful classifiers with more training data (bias-variance trade-off) Adapted from D. Hoiem

More Related