1 / 34

Linear Models

Linear Models. Tony Dodd. Overview. Linear models. Parameter estimation. Linear in the parameters. Classification. The nonlinear bits. Linear models. Linear model has general form where is the th component of input . Assume and therefore is the bias.

elavoie
Download Presentation

Linear Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Linear Models Tony Dodd

  2. Overview • Linear models. • Parameter estimation. • Linear in the parameters. • Classification. • The nonlinear bits. An Overview of State-of-the-Art Data Modelling

  3. Linear models • Linear model has general form where is the th component of input . • Assume and therefore is the bias. • Can represent lines and planes. • Should ALWAYS try a linear model first! An Overview of State-of-the-Art Data Modelling

  4. Parameter estimation • Least squares estimation. • Choose parameters that minimise • Unique minimum… • Optimum when noise is Gaussian. An Overview of State-of-the-Art Data Modelling

  5. Least squares cost function An Overview of State-of-the-Art Data Modelling

  6. Least squares parameters • Define the design matrix • Then the optimal parameters given by An Overview of State-of-the-Art Data Modelling

  7. Example An Overview of State-of-the-Art Data Modelling

  8. Example An Overview of State-of-the-Art Data Modelling

  9. Example An Overview of State-of-the-Art Data Modelling

  10. How can we generalise this? • Consider instead • Where is a nonlinear function of the inputs. • Nonlinear transform of the inputs and then form a linear model (more tomorrow). An Overview of State-of-the-Art Data Modelling

  11. Linear in the parameters • A nonlinear model that is often called linear. • Can apply simple estimation to the parameters. • But… it is nonlinear in the basis functions. An Overview of State-of-the-Art Data Modelling

  12. Parameter estimation • Define the design matrix • Then the optimal parameters given by An Overview of State-of-the-Art Data Modelling

  13. Example An Overview of State-of-the-Art Data Modelling

  14. Example An Overview of State-of-the-Art Data Modelling

  15. Example An Overview of State-of-the-Art Data Modelling

  16. Example – how does it work? An Overview of State-of-the-Art Data Modelling

  17. Example – how does it work? An Overview of State-of-the-Art Data Modelling

  18. Example – how does it work? To get the function estimate Add all these together An Overview of State-of-the-Art Data Modelling

  19. Example – when it all goes wrong More on this later. An Overview of State-of-the-Art Data Modelling

  20. Linear classification How do we apply linear models to classification – output is now categorical? • Discriminant analysis. • Probit analysis. • Log-linear regression. • Logistic regression. An Overview of State-of-the-Art Data Modelling

  21. Logistic regression • A regression model for Bernoulli-distributed targets. • Form the linear model where An Overview of State-of-the-Art Data Modelling

  22. Can we generalise it? • Instead of use a linear in the parameters model An Overview of State-of-the-Art Data Modelling

  23. Parameter estimation • Maximum likelihood. • Maximise the probability of getting the observed results given the parameters. • Although unique minimum need to use iterative techniques (no closed form solution). An Overview of State-of-the-Art Data Modelling

  24. Example An Overview of State-of-the-Art Data Modelling

  25. Example An Overview of State-of-the-Art Data Modelling

  26. Example An Overview of State-of-the-Art Data Modelling

  27. Example An Overview of State-of-the-Art Data Modelling

  28. Example – class probabilities An Overview of State-of-the-Art Data Modelling

  29. But… An Overview of State-of-the-Art Data Modelling

  30. Basis function optimisation Need to estimate: • Type of basis functions. • Number of basis functions. • Positions of basis functions. These are nonlinear problems – difficult! An Overview of State-of-the-Art Data Modelling

  31. Types of basis functions • Usually choose a favourite! • Examples include: Polynomials: Gaussians: … An Overview of State-of-the-Art Data Modelling

  32. Number of basis functions • How many basis functions? • Slowly increase number until overfit data. • Exploratory vs optimal. • More on this in the next talk. An Overview of State-of-the-Art Data Modelling

  33. Positions of basis functions • This is really difficult! • One easy possibility is to put one basis function on each data point. • Uniform grid (but curse of dimensionality). • Advantage of global basis functions e.g. polynomials – don’t need to optimise positions. An Overview of State-of-the-Art Data Modelling

  34. Concluding remarks • Always try a linear model first. • Can make nonlinear in the input but linear in the parameters. • But becomes nonlinear optimisation. • Is least squares/maximum likelihood the best way? An Overview of State-of-the-Art Data Modelling

More Related