1 / 37

CS 59000 Statistical Machine learning Lecture 7

CS 59000 Statistical Machine learning Lecture 7. Yuan (Alan) Qi Purdue CS Sept. 16 2008. Acknowledgement: Sargur Srihari’s slides. Outline . Review of noninformative priors, nonparametric methods, and nonlinear basis functions Regularized regression Bayesian regression Equivalent kernel

stuart-greene
Download Presentation

CS 59000 Statistical Machine learning Lecture 7

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CS 59000 Statistical Machine learningLecture 7 Yuan (Alan) Qi Purdue CS Sept. 16 2008 Acknowledgement: Sargur Srihari’s slides

  2. Outline Review of noninformative priors, nonparametric methods, and nonlinear basis functions Regularized regression Bayesian regression Equivalent kernel Model Comparison

  3. The Exponential Family (1) where ´ is the natural parameter and so g(´) can be interpreted as a normalization coefficient.

  4. Property of Normalization Coefficient From the definition of g(´) we get Thus

  5. Conjugate priors For any member of the exponential family, there exists a prior Combining with the likelihood function, we get Prior corresponds to º pseudo-observations with value Â.

  6. Noninformative Priors (1) With little or no information available a-priori, we might choose a non-informative prior. ¸ discrete, K-nomial : ¸2[a,b] real and bounded: ¸ real and unbounded: improper! A constant prior may no longer be constant after a change of variable; consider p(¸) constant and ¸=´2:

  7. Noninformative Priors (2) Translation invariant priors. Consider For a corresponding prior over ¹, we have for any A and B. Thus p(¹) = p(¹ { c) and p(¹) must be constant.

  8. Noninformative Priors (4) Scale invariant priors. Consider and make the change of variable For a corresponding prior over ¾, we have for any A and B. Thus p(¾) / 1/¾ and so this prior is improper too. Note that this corresponds to p(ln¾) being constant.

  9. Nonparametric Methods (1) Parametric distribution models are restricted to specific forms, which may not always be suitable; for example, consider modelling a multimodal distribution with a single, unimodal model. Nonparametric approaches make few assumptions about the overall shape of the distribution being modelled.

  10. Nonparametric Methods (2) Histogram methods partition the data space into distinct bins with widths ¢i and count the number of observations, ni, in each bin. Often, the same width is used for all bins, ¢i = ¢. ¢ acts as a smoothing parameter. In a D-dimensional space, using M bins in each dimen-sion will require MD bins!

  11. Nonparametric Methods (3) If the volume of R, V, is sufficiently small, p(x) is approximately constant over R and Thus Assume observations drawn from a density p(x) and consider a small region R containing x such that The probability that K out of N observations lie inside R is Bin(KjN,P ) and if N is large

  12. Nonparametric Methods (5) To avoid discontinuities in p(x), use a smooth kernel, e.g. a Gaussian Any kernel such that will work. h acts as a smoother.

  13. K-Nearest-Neighbours for Classification (1) Given a data set with Nk data points from class Ck and , we have and correspondingly Since , Bayes’ theorem gives

  14. K-Nearest-Neighbours for Classification (2) K = 1 K = 3

  15. Basis Functions

  16. Examples of Basis Functions (1)

  17. Maximum Likelihood Estimation (1)

  18. Maximum Likelihood Estimation (2)

  19. Sequential Estimation

  20. Regularized Least Squares

  21. More Regularizers

  22. Visualization of Regularized Regression

  23. Bayesian Linear Regression

  24. Posterior Distributions of Parameters

  25. Predictive Posterior Distribution

  26. Examples of PredictiveDistribution

  27. Question Suppose we use Gaussian basis functions. What will happen to the predictive distribution if we evaluate it at places far from all training data points?

  28. Equivalent Kernel Given Predictive mean is where

  29. Equivalent kernel Basis Function: Equivalent kernel: Gaussian Polynomial Sigmoid

  30. Covariance between two predictions Predictive mean at nearby points will be highly correlated, whereas for more distant pairs of points the correlation will be smaller.

  31. Bayesian Model Comparison Suppose we want to compare models . Given a training set , we compute Model evidence (also known as marginal likelihood): Bayes factor:

  32. Likelihood, Parameter Posterior & Evidence Likelihood and evidence Parameter posterior distribution and evidence

  33. Crude Evidence Approximation Assume posterior distribution is centered around its mode

  34. Evidence penalizes over-complex models Given M parameters Maximizing evidence leads to a natural trade-off between data fitting & model complexity.

  35. Evidence Approximation & Empirical Bayes Approximating the evidence by maximizing marginal likelihood. Where hyperparameters maximize the evidence . Known as Empirical Bayes or type2 maximum likelihood

  36. Model Evidence and Cross-Validation Fitting polynomial regression models Root-mean-square error Model evidence

  37. Next class Linear Classification

More Related