Jim Ramsay McGill University

1. Basis Basics Jim Ramsay McGill University

2. Overview • What are basis functions? • What properties should they have? • How are they usually constructed? • Fourier bases. • B-spline bases: Lots of detail here. • Wavelet bases. • Advice, wrap-up, and beyond.

3. What are basis functions? • We need flexible method for constructing a function f(t) that can track local curvature. • We pick a system of Kbasis functions φk(t), and call this thebasisfor f(t). • We express f(t) as a weighted sum of these basis functions: f(t) =a1φ1(t) + a2φ2(t) + … + aKφK(t) The coefficients a1, … , aK determine the shape of the function.

4. What do we want from basis functions? • Fast computation of individual basis functions. • Flexible: can exhibit the required curvature where needed, but also be nearly linear when appropriate. • Fast computation of coefficients ak: possible if matrices of values are diagonal, banded or sparse. • Differentiable as required: We make lots of use of derivatives in functional data analysis. • Constrained as required, such as periodicity, positivity, monotonicity, asymptotes and etc.

5. What are some commonly used basis functions? • Powers: 1, t, t2, and so on. They are the basis functions for polynomials. These are not very flexible, and are used only for simple problems. • Fourier series: 1, sin(ωt), cos(ωt), sin(2ωt), cos(2ωt), and so on for a fixed known frequency ω. These are used for periodic functions. • B-spline functions: These have now more or less replaced polynomials for non-periodic problems. More explanation follows.

6. How do we construct basis systems? • We start with a prototype basis function φ(t). Let’s call it the mother function. • We apply three operations to it: • Lateral shiftφ(t+a) to focus fitting power near a position a. • Scale changeφ(bt), to increase resolving power, or the capacity for local curvature. • Smoothing, to increase differentiability and smoothness.

7. Can I see an example? Consider the Fourier series: • Mother functions: φ(t) = sin(ωt), and φ(t) = 1 • Lateral Shift: sin[ω(t+π/2)] = cos(ωt) • Scale change: sin(kωt), cos(kωt), k=1,2,… • Smoothing: None; these are already infinitely differentiable functions. • Strengths: Orthogonality makes computing coefficients very fast • Limitations: Fourier series are only natural if f(t) is periodic with period ω.

8. Five Fourier Basis Functions • φ1(t) = 1 • A sine/cosine pair with period 1: Captures low frequency variation • A sine/cosine pair with period ½: Captures higher frequency variation

9. What about polynomials? • The mother function is φ(t) = 1. • Smoothing: Multiply by t to increase both differentiability and curvature. • No shift or scaling: Polynomials are invariant with respect to shift and scaling. • Strengths: Okay for very simple curves such as linear or slight curvature. • Limitations: Just not flexible enough. We can’t focus fitting power on a specific location.

10. What basis should I use for non-periodic functions? Spline bases have nearly everything: • Infinite and independent control over shift, scale, and smoothness. • Potential to model sharp changes in f(t) as well as its smooth variation. • As fast as polynomials to evaluate. • Band-structured matrix of values.

11. How do B-splines work? • Start with a discrete mesh of values, called knots: ξ0, ξ1, …, ξL. • Space out the knots where you don’t need much resolution, make them dense where you need a lot of resolution.

12. Mother function is the box function over [ξ0, ξ1]: φ(t) = 1, ξ0 ≤ t < ξ1, and 0 otherwise. • Shift: just shift to another interval. • Scale: determined by spacing between knots.

13. What about smoothness? • Smoothness control is the heart of spline technology. • It is defined by a recursion relation. • The recursion relation uses spline basis functions at a given smoothness level to make new ones at the next higher smoothness level.

14. B-spline notation The spline basis system we use is the B-spline system. • To keep notation simple, let the knots be the natural numbers j = 0, 1, … • We need two indices for a B-spline basis function Bjm(t):j to indicate the knot where it starts, and m to indicate its order or degree of smoothness. • For our box functions, m = 1.

15. B-splines of order 2 are tent functions, starting at a knot, rising linearly to 1 at the next knot, and decaying linearly to 0 two knots over. • They are continuous. Order 2 implies a continuous derivative of order 0. • Order 2 knots are piecewise linear. • What about order 3 B-splines?

16. The recursion for integer knots For the B-spline function of order 2 beginning at 0, the recursion is

17. In the first term, the mother box function B01(t) is multiplied by t, and is therefore t itself. • In the second, the mother box is shifted to the right and multiplied by 2-t. • The second term amounts to a shift and reflection of the first term.

18. Constructing a Tent B02(t) from Two Boxes B01(t) and B11(t)

19. Constructing Order 3 B03(t) from Two Tent Functions

20. Multiplying by (t-j) increases the polynomial degree of the B-spline in the first term by one. • Multiplying (m+j-t) does the same thing for the reflected and shifted B-spline in the second term. • The result is a new B-spline of one degree or order higher, spread over an additional interval, and having an one more level of continuous derivatives. • Order 3 B-splines are piece-wise quadratic, spread over 3 intervals, and have a continuous first derivative.

21. How do we get the extra smoothness? • tB01(t) and (2-t)B11(t) have the same values at knot 1. The sum is continuous. • tB02(t) and (3-t)B12(t) have the same derivative values at knots 1 and 2. The sum has a continuous first derivative. • Each time we multiply by t, and then shift and reflect the result, we add another order of smooth derivative.

22. Suppose I don’t want so much smoothness at a fixed point? • You can have multiple knots at a point. • For each additional knot, the spline function will have one less derivative at that knot. • An order m Bspline with m knots at a point can be discontinuous at that point. • Most spline software places m knots at each of the boundaries.

23. How many B-spline basis functions am I using? • When all interior knots are unique, the total number of B-spline basis functions is equal to the number of interior knots plus the order. • When there are no interior knots, splines become polynomials: The order equals the number of basis functions.

24. Wavelets • Wavelets take this shift/scale/smooth trio of operations even further. • They define basis functions organized into orthogonal levels of resolution. • The resulting multi-resolution analysis is one of the most exciting developments in mathematics since Fourier’s series.

25. Some advice • If you have periodic data, you’ll probably use Fourier series. • If you don’t, you’ll probably use B-splines. • Especially if you need to look at or work with derivatives. • If you want a derivative of order m, fit the data with a spline of order at least m+2, although order 2m has some advantages.

26. Where have we been? • Linear combinations of basis functions are a convenient way to model functions. • The more basis functions we use, the more complex the fitted function can be. • Most basis function systems involve turning three knobs, marked shift, scale, and smooth, to modify a mother function. • The best systems permit independent control of these features. • Like splines and wavelets.

27. Where do we go from here? • Although we can control the resolution of a spline fit by the number of spline basis functions and the spacing of knots, we often need easier more continuous control over smoothness. • Roughness penalties can give us the additional control that we need.