1 / 30

Bias-Variance Trade-Off in Classifier Selection and Optimization

This course outline covers principles of generalization, survey of classifiers, discussion of the Rosch Pipeline for Prediction Imagery Representation Classifier, and topics such as bias and variance errors, overfitting, and the need for validation sets in reducing variance in models. It discusses the impact of features on bias and variance, how to measure complexity using VC dimension, and strategies to reduce variance through model parameterization and regularization. The course also delves into various classification methods like generative, discriminative, ensemble, instance-based, and unsupervised methods, highlighting their objectives, parameterization, training process, and inference. References touch on resources such as Tom Mitchell's "Machine Learning" and Christopher Bishop's "Neural Networks for Pattern Recognition" for further reading.

ckessel
Download Presentation

Bias-Variance Trade-Off in Classifier Selection and Optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Classification Derek Hoiem CS 598, Spring 2009 Jan 27, 2009

  2. Outline • Principles of generalization • Survey of classifiers • Project discussion • Discussion of Rosch

  3. Pipeline for Prediction Imagery Representation Classifier Predictions

  4. Free Lunch Theorem

  5. Bias and Variance Error High Bias Low Variance Complexity Low Bias High Variance

  6. Overfitting • Need validation set • Validation set not same as test set

  7. Bias-Variance View of Features • More compact = lower variance, potentially higher bias • More features = higher variance, lower bias • More independence among features = simpler classifier  lower variance

  8. How to reduce variance • Parameterize model E.g., linear vs. piecewise

  9. How to measure complexity? • VC dimension Upper bound on generalization error Training error + N: size of training set h: VC dimension : 1-probability

  10. How to reduce variance • Parameterize model • Regularize

  11. How to reduce variance • Parameterize model • Regularize • Increase number of training examples

  12. Effect of Training Size Error Number of Training Examples

  13. Risk Minimization • Margins x x x x x x x o x o o o o x2 x1

  14. Classifiers • Generative methods • Naïve Bayes • Bayesian Networks • Discriminative methods • Logistic Regression • Linear SVM • Kernelized SVM • Ensemble methods • Randomized Forests • Boosted Decision Trees • Instance based • K-nearest neighbor • Unsupervised • Kmeans

  15. Components of classification methods • Objective function • Parameterization • Regularization • Training • Inference

  16. Classifiers: Naïve Bayes • Objective • Parameterization • Regularization • Training • Inference y x1 x2 x3

  17. Classifiers: Logistic Regression • Objective • Parameterization • Regularization • Training • Inference

  18. Classifiers: Linear SVM • Objective • Parameterization • Regularization • Training • Inference x x x x x x x o x o o o o x2 x1

  19. Classifiers: Linear SVM • Objective • Parameterization • Regularization • Training • Inference x x x x x x x o x o o o o x2 x1

  20. Classifiers: Linear SVM • Objective • Parameterization • Regularization • Training • Inference Needs slack x x o x x x x x o x o o o o x2 x1

  21. Classifiers: Kernelized SVM • Objective • Parameterization • Regularization • Training • Inference x x o o o x x x1 x x o o x12 o x x x1

  22. Classifiers: Decision Trees • Objective • Parameterization • Regularization • Training • Inference x x x x x x o x o x o o o o x2 x1

  23. Ensemble Methods: Boosting figure from Friedman et al. 2000

  24. Boosted Decision Trees High in Image? Gray? Yes No Yes No Smooth? Green? High in Image? Many Long Lines? … Yes Yes No Yes No Yes No No Blue? Very High Vanishing Point? Yes No Yes No P(label | good segment, data) Ground Vertical Sky [Collins et al. 2002]

  25. Boosted Decision Trees • How to control bias/variance trade-off • Size of trees • Number of trees

  26. K-nearest neighbor • Objective • Parameterization • Regularization • Training • Inference x x o x x x x o x o x o o o o x2 x1

  27. Clustering + x + o + + x x + x + + x + + o x x + o + o + o x2 x2 x1 x1

  28. References • General • Tom Mitchell, Machine Learning, McGraw Hill, 1997 • Christopher Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995 • Adaboost • Friedman, Hastie, and Tibshirani, “Additive logistic regression: a statistical view of boosting”, Annals of Statistics, 2000 • SVMs • http://www.support-vector.net/icml-tutorial.pdf

  29. Project ideas?

  30. Discussion of Rosch

More Related