Basis Expansion and Regularization

1. Basis Expansion and Regularization Prof.Liqing Zhang Dept. Computer Science & Engineering, Shanghai Jiaotong University

2. Outline • Piece-wise Polynomials and Splines • Smoothing Splines • Automatic Selection of the Smoothing Parameters • Nonparametric Logistic Regression • Multidimensional Splines • Regularization and Reproducing Kernel Hilbert Spaces • Wavelet Smoothing Basis Expansion and Regularization

3. Piece-wise Polynomials and Splines • Linear basis expansion • Some basis functions that are widely used Basis Expansion and Regularization

4. Regularization • Three approaches for controlling the complexity of the model. • Restriction • Selection • Regularization: Basis Expansion and Regularization

5. Piecewise Polynomials and Splines Basis Expansion and Regularization

6. Piecewise Cubic Polynomials • Increasing orders of continuity at the knots. • A cubic spline with knots at and : • Cubic spline truncated power basis Basis Expansion and Regularization

7. Piecewise Cubic Polynomials • An order-M spline with knots , j=1,…,K is a piecewise-polynomial of order M, and has continuous derivatives up to order M-2. • A cubic spline has M=4. • Truncated power basis set: Basis Expansion and Regularization

8. Natural boundary constraints Linear Cubic Cubic Linear Natural cubic spline • Natural cubic spline adds additional constraints, namely that the function is linear beyond the boundary knots. Basis Expansion and Regularization

9. B-spline • The augmented knot sequenceτ: • Bi,m(x), the i-th B-spline basis function of order m for the knot-sequence τ, m≤M.

10. The sequence of B-spline up to order 4 with ten knots evenly spaced from 0 to 1 The B-spline have local support; they are nonzero on an interval spanned by M+1 knots. B-spline Basis Expansion and Regularization

11. Boundary Effect in Variances

12. Smoothing Splines • Base on the spline basis method: • So , is the noise. • Minimize the penalized residual sum of squares is a fixed smoothing parameter f can be any function that interpolates the data the simple least squares line fit Basis Expansion and Regularization

13. Smoothing Splines • The solution is a natural spline: • Then the criterion reduces to: • where • So the solution: • The fitted smoothing spline: Basis Expansion and Regularization

14. Smoothing Splines • The relative change in bone mineral density measured at the spline in adolescents • Separate smoothing splines fit the males and females, • 12 degrees of freedom 脊骨BMD—骨质密度 Basis Expansion and Regularization

15. Smoothing Matrix • the N-vector of fitted values • The finite linear operator — the smoother matrix • Compare with the linear operator in the LS-fitting: M cubic-spline basis functions, knot sequence ξ • Similarities and differences: • Both are symmetric, positive semidefinite matrices • idempotent（幂等的) ; shrinking • rank: Basis Expansion and Regularization

16. Smoothing Matrix • Effective degrees of freedomof a smoothing spline • in the Reinsch form: • Since , solution: • is symmetric and has a real eigen-decomposition • is the corresponding eigenvalue of K Basis Expansion and Regularization

17. Smoothing spline fit of ozone(臭氧) concentration versus Daggot pressure gradient. • Smoothing parameter df=5 and df=10. • The 3rd to 6th eigenvectors of the spline smoothing matrices

18. The smoother matrix for a smoothing spline is nearly banded, indicating an equivalent kernel with local support. Basis Expansion and Regularization

19. Bias-Variance Tradeoff • Example: • For • The diagonal contains the pointwise variances at the training • Bias is given by • is the (unknown) vector of evaluations of the true f Basis Expansion and Regularization

20. df=9, bias slight, variance not increased appreciably • df=15, over learning, standard error widen Bias-Variance Tradeoff • df=5, bias high, standard error band narrow Basis Expansion and Regularization

21. The EPE and CV curves have the a similar shape. And, overall the CV curve is approximately unbiased as an estimate of the EPE curve Bias-Variance Tradeoff • The integrated squared prediction error (EPE) combines both bias and variance in a single summary: • N fold (leave one) cross-validation: Basis Expansion and Regularization

22. Outline • Piece-wise Polynomials and Splines • Smoothing Splines • Automatic Selection of the Smoothing Parameters • Nonparametric Logistic Regression • Multidimensional Splines • Regularization and Reproducing Kernel Hilbert Spaces • Wavelet Smoothing Basis Expansion and Regularization

23. Logistic Regression • Logistic regression with a single quantitative input X • The penalized log-likelihood criterion Basis Expansion and Regularization

24. Multidimensional Splines • Tensor product basis • The M1×M2 dimensional tensor product basis • , basis function for coordinate X1 • , basis function for coordinate X2 Basis Expansion and Regularization

25. Tenor product basis of B-splines, some selected pairs Basis Expansion and Regularization

26. Multidimensional Splines • High dimension smoothing Splines • J is an appropriate penalty function a smooth two-dimensional surface, a thin-plate spline. • The solution has the form Basis Expansion and Regularization

27. Multidimensional Splines • The decision boundary of an additive logistic regression model. Using natural splines in each of two coordinates. • df = 1 +(4-1) + (4-1) = 7 Basis Expansion and Regularization

28. Multidimensional Splines • The results of using a tensor product of natural spline basis in each coordinate. • df = 4 x 4 = 16 Basis Expansion and Regularization

29. Multidimensional Splines • A thin-plate spline fit to the heart disease data. • The data points are indicated, as well as the lattice of points used as knots. Basis Expansion and Regularization

30. Basis Expansion and Regularization

31. Outline • Piece-wise Polynomials and Splines • Smoothing Splines • Automatic Selection of the Smoothing Parameters • Nonparametric Logistic Regression • Multidimensional Splines • Regularization and Reproducing Kernel Hilbert Spaces • Wavelet Smoothing Basis Expansion and Regularization

32. Reproducing Kernel Hilbert space • A regularization problems has the form: • L(y,f(x)) is a loss-function. • J(f) is a penalty functional, and H is a space of functions on which J(f) is defined. • The solution • span the null space of the penalty functional J Basis Expansion and Regularization

33. Spaces of Functions Generated by Kernel • Important subclass are generated by the positive kernel K(x,y). • The corresponding space of functions Hk is called reproducing kernel Hilbert space. • Suppose that K has an eigen-expansion • Elements of H have an expansion Basis Expansion and Regularization

34. Spaces of Functions Generated by Kernel • The regularization problem become • The finite-dimension solution(Wahba,1990) • Reproducing properties of kernel function

35. Spaces of Functions Generated by Kernel • The penalty functional • The regularization function reduces to a finite-dimensional criterion • K is NxN matrix Basis Expansion and Regularization

36. RKHS • Penalized least squares • The solution : • The fitted values: • The vector of N fitted value is given by

37. Example of RKHS • Polynomial regression • Suppose , M huge • Given • Loss function: • The penalty polynomial regression: Basis Expansion and Regularization

38. Penalized Polynomial Regression • Kernel: has eigen-functions • E.g. d=2, p=2: M=6 • The penalty polynomial regression: Basis Expansion and Regularization

39. RBF kernel & SVM kernel • Gaussian Radial Basis Functions • Support Vector Machines Basis Expansion and Regularization

40. Denoising Basis Expansion and Regularization

41. Wavelet smoothing • Another type of bases——Wavelet bases • Wavelet bases are generated by translations and dilations of a single scaling function . • If ,then generates an orthonormal basis for functions with jumps at the integers. • form a space called reference space V0 • The dilations form an orthonormal basis for a space V1 V0 • Generally, we have Basis Expansion and Regularization

42. Basis Expansion and Regularization

43. Basis Expansion and Regularization

44. Wavelet smoothing • Wavelet basis on W0 : • Wavelet basis on Wj : • The symmlet-p wavelet: • A support of 2p-1 consecutive intervals. • p vanishing moments: Basis Expansion and Regularization

45. Basis Expansion and Regularization

46. Wavelet smoothing • The L2 space dividing • Mother wavelet generate function form an orthonormal basis for W0. Likewise form a basis for Wj. Basis Expansion and Regularization

47. Noise Reduction Basis Expansion and Regularization

48. Basis Expansion and Regularization

49. Adaptive Wavelet Filtering • Wavelet transform: • y: response vector, W: NxN orthonormal wavelet basis matrix • Stein Unbiased Risk Estimation (SURE) • The solution: • Fitted function is given by inverse wavelet transform: , LS coefficients truncated to 0 Basis Expansion and Regularization

50. Parameter Selection • Simple choice for • If random variables are white noise with variance σ, the expected maximum of is approximately . • Hence all coefficients below are likely to be noise and are set to zero. Basis Expansion and Regularization