1 / 17

Basis Expansions and Regularization

Explore advanced basis expansions and regularization techniques for linear models based on Chapter 5 of Hastie, Tibshirani, and Friedman. Learn about cubic splines, natural cubic splines, smoothing splines, nonparametric logistic regression, and thin-plate splines. These methods help prevent overfitting and improve model performance. Dive into the theoretical foundations and practical applications of these techniques.

schmelingm
Download Presentation

Basis Expansions and Regularization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Basis Expansions and Regularization Based on Chapter 5 of Hastie, Tibshirani and Friedman

  2. Basis Expansions for Linear Models Here the hm’s might be: • hm(X)=Xm, m=1,…,p recovers the original model • hm(X)=Xj2 orhm(X)= XjXk • hm(X)=I(LmXk Um),

  3. “knots”

  4. Regression Splines Bottom left panel uses: Number of parameters = (3 regions) X (2 params per region) - (2 knots X 1 constraint per knot) = 4

  5. cubic spline

  6. Cubic Spline continuous first and second derivatives Number of parameters = (3 regions) X (4 params per region) - (2 knots X 3 constraints per knot) = 6Knot discontinuity essentially invisible to the human eye

  7. Natural Cubic Spline Adds a further constraint that the fitted function is linear beyond the boundary knots A natural cubic spline model with K knots is represented by K basis functions: Each of these basis functions has zero 2nd and 3rd derivative outside the boundary knots

  8. Natural Cubic Spline Models Can use these ideas in, for example, regression models. For example, if you use 4 knots and hence 4 basis functions per predictor variable, then simply fit logistic regression model with four times the number of predictor variables…

  9. Smoothing Splines Consider this problem: among all functions f(x) with two continuous derivatives, find the one that minimizes the penalized residual sum of squares: smoothing parameter =0 : f can be any function that interpolates the data =infinity : least squares line

  10. Smoothing Splines Theorem: The unique minimizer of this penalized RSS is a natural cubic spline with knots at the unique values of xi , i=1,…,N Seems like there will be N features and presumably overfitting of the data. But,… the smoothing term shrinks the model towards the linear fit This is a generalized ridge regression Can show that where K does not depend on 

  11. Nonparametric Logistic Regression Consider logistic regression with a single x: and a penalized log-likelihood criterion: Again can show that the optimal f is a natural spline with knots at the datapoint Can use Newton-Raphson to do the fitting.

  12. Thin-Plate Splines The discussion up to this point has been one-dimensional. The higher-dimensional analogue of smoothing splines are “thin-plate splines.” In 2-D, instead of minimizing: minimize:

  13. Thin-Plate Splines The solution has the form: a type of “radial basis function”

More Related