1 / 40

Overview of Supervised Learning

Overview of Supervised Learning. Outline. Linear Regression and Nearest Neighbors method Statistical Decision Theory Local Methods in High Dimensions Statistical Models, Supervised Learning and Function Approximation Structured Regression Models Classes of Restricted Estimators

stefan
Download Presentation

Overview of Supervised Learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Overview of Supervised Learning

  2. Outline • Linear Regression and Nearest Neighbors method • Statistical Decision Theory • Local Methods in High Dimensions • Statistical Models, Supervised Learning and Function Approximation • Structured Regression Models • Classes of Restricted Estimators • Model Selection and Bias Overview of Supervised Learning

  3. Notation • X: inputs, feature vector, predictors, independent variables. Generally X will be a vector of p values. Qualitative features are coded in X. • Sample values of X generally in lower case; xi is i-th of N sample values. • Y: output, response, dependent variable. • Typically a scalar, can be a vector, of real values. Again yi is a realized value. • G: a qualitative response, taking values in a discrete set G; e.g. G={ survived, died }. We often code G via a binary indicator response vector Y. Overview of Supervised Learning

  4. Problem • 200 points generated in IR2 from a unknown distribution; 100 in each of two classes G={ GREEN, RED }. • Can we build a rule to predict the color of the future points? Overview of Supervised Learning

  5. Linear regression • Code Y=1 if G=RED, else Y=0. • We model Y as a linear function of X: • Obtain bby least squares, by minimizing the quadratic criterion: • Given an model matrix X and a response vector y, Overview of Supervised Learning

  6. Linear regression Overview of Supervised Learning

  7. Linear regression • Figure 2.1: A Classification example in two dimensions. The classes are coded as a binary variable (GREEN=0, RED=1) and then fit by linear regression. The line is the decision boundary defined by . The red shaded region denotes that part of input space classified as RED ,while the green region is classified as GREEN. Overview of Supervised Learning

  8. Possible scenarios Overview of Supervised Learning

  9. K-Nearest Neighbors Overview of Supervised Learning

  10. K-Nearest Neighbors • Figure 2.2: The same classification example in two dimensions as in Figure 2.1. The classes are coded as a binary variable (GREEN=0, RED=1) and the fit by 15-nearest-neighbor. • The predicted class is hence chosen by majority vote amongst the 15-nearest neighbors. Overview of Supervised Learning

  11. K-Nearest Neighbors • Figure 2.3: The same classification example are coded as a binary variable ( GREEN=0, RED=1), and then predicted by 1-nearest-neighbor classification. Overview of Supervised Learning

  12. Linear regression vs. k-NN Overview of Supervised Learning

  13. Linear regression vs. k-NN • Figure 2.4: Misclassification curves for the simulation example above. a test sample of size 10,000 was used. The red curves are test and the green are training error for k-NN classification. The results for linear regression are the bigger green and red dots at three degrees of freedom. The purple line is the optimal Bayes Error Rate. Overview of Supervised Learning

  14. Other Methods Overview of Supervised Learning

  15. Statistical decision theory Overview of Supervised Learning

  16. 回归函数 Overview of Supervised Learning

  17. Overview of Supervised Learning

  18. Overview of Supervised Learning

  19. Bayes Classifier Overview of Supervised Learning

  20. Bayes Classifier • Figure 2.5: The optimal Bayes decision boundary for the simulation example above. • Since the generating density is known for each class, this boundary can be calculated exactly. Overview of Supervised Learning

  21. Curse of dimensionality Overview of Supervised Learning

  22. Overview of Supervised Learning

  23. Overview of Supervised Learning

  24. Overview of Supervised Learning

  25. Linear Model • Linear Model • Linear Regression • Test error Overview of Supervised Learning

  26. Curse of dimensionality Overview of Supervised Learning

  27. Overview of Supervised Learning

  28. Statistical Models Overview of Supervised Learning

  29. Supervised Learning Overview of Supervised Learning

  30. Two Types of Supervised Learning Overview of Supervised Learning

  31. Learning Classification Models Overview of Supervised Learning

  32. Learning Regression Models Overview of Supervised Learning

  33. Function Approximation Overview of Supervised Learning

  34. Function Approximation • Figure 2.10: Least squares fitting of a function of two inputs. The parameters of fθ(x) are chosen so as to minimize the sum-of-squared vertical errors. Overview of Supervised Learning

  35. Function Approximation • More generally, Maximum Likelihood Estimation provides a natural basis for estimation. • E.g. multinomial Overview of Supervised Learning

  36. Structured Regression Models Overview of Supervised Learning

  37. Classes of Restricted Estimators Overview of Supervised Learning

  38. Model Selection & the Bias-Variance Tradeoff Overview of Supervised Learning

  39. Model Selection & the Bias-Variance Tradeoff • Test and training error as a function of model complexity. Overview of Supervised Learning

  40. Page 27 • Ex 2.1; 2.2; 2.4； 2.6 Overview of Supervised Learning

More Related