CS 312: Algorithm Analysis

1. This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License. CS 312: Algorithm Analysis Lecture #7: Recurrence Relations a.k.a. Difference Equations Slides by: Eric Ringger, with contributions from Mike Jones, Eric Mercer, Sean Warnick

2. Announcements • HW #4 Due Now • Project #2 • Early: next Wednesday • Due: Friday 2/1 • Much more challenging than Project #1– plan accordingly • Career Fair!

3. Objectives • Goal: Analyze Divide and Conquer recursive algorithms using recurrence relations (RRs) • Also known as Difference Equations • Working up to such RRs • Today: Focus on a special type of RRs • Homogeneous • Linear • Constant Coefficients • Leading up to a proof of the Master Theorem

4. c2g(n) f(n) c1g(n) n0 Recall: Analysis Framework Given a problem, • Identify platform’s elementary operations (sometimes implicit) • Formulate solution as an algorithm • Define the measure of the input size • Measure time efficiency by counting the number of times an elementary operation is executed • Measure space efficiency by counting the number of memory units consumed by the algorithm • The efficiency of some algorithms may differ significantly for inputs of the same size. • Distinguish among worst-case, average-case, and best-case efficiencies. Choose one. • Plot efficiency vs. input size • Establish order of growth. Use asymptotic notation. Where is the difficulty for Analyzing Recursive Functions?

5. Recurrence Relations • Theory to aid us in analyzing recursive algorithms • Given a sequence of (real) numbers • A recurrence relation (RR) is an equation (a rule) expressing the value in terms of earlier (usually neighboring) values. • Examples: • The RR describes a function (e.g., or ) by such a rule instead of by a list of its values! • is the index variable(representing, e.g., moment in time, size of an input to an algorithm, etc.)

6. Example: Compute Factorial function N! Recursive Algorithms • Assume 32-bit integers • Define input size: _____ • Define elementary operations: • Focus on worst-case • functionFactorial(N) • if N=0 return 1 • else return Factorial(N-1)*N What sequence is generated by this recurrence relation? In order to answer that question we need an initial condition! N C(N) Now we can build a table of values: How long to compute C(1,000,000)? Want: C(N) in closed form for quick computation.

7. Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one. Recursive Algorithms How to solve?

8. Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one. Recursive Algorithms

9. Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one. Recursive Algorithms

10. Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one. Recursive Algorithms

11. Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one. Recursive Algorithms

12. Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one. Recursive Algorithms Can you figure out an explicit (closed form) formula for this sequence? • As before, build a table, recognize the pattern • OR • Use substitution, recognize the pattern • OR • Appeal to theory of recurrence relations n C(n)

13. y(k) k 1 2 3 4 5 6 7 8 9 Substitution

14. Recurrence Relations The most general form for our purposes: RR is linear when it can be written as: where: • is arbitrary function of • Coefficients are functions of independent of for all in • Order is finite

15. Constant Coefficients • If the coefficients ( do not depend on , the equation is said • to have constant coefficients • to be time-invariant. • LTI = “linear, time-invariant”

16. Forcing Function • Thefunction is called • the forcing function • the forcingterm • the drivingterm • the systeminput • or simply the right-hand side

17. Homogeneous • A solution of the RR is a function that satisfies the recurrence relation: • A recurrence relation is said to be homogeneous if is a solution of the equation (for all ). • Consequence: • We’ll focus on this type today.

18. y(k) k 1 2 3 4 5 6 7 8 9 What kind of RR is this?

19. Terminology Review Note the alternate notations.

20. Terminology Review Note the alternate notations.

21. Existence and Uniqueness Theorem For an arbitrary real-valued function , let a difference equation of the form be defined over a sequence of consecutive integer values of . Then the equation has one and only one solution corresponding to each specification of the initial values (i.e., initial conditions) . Proof (sketch): Suppose the initial values are specified. Then the value of can be uniquely determined simply by evaluating the function. The proof then proceeds by induction for each consecutive value of .

22. Starting Point • The theory of recurrence relations (and linear systems) gives us a basic solution: For some value .

23. Goal: find solution tn Example

24. Goal: find solution tn Example

25. General Solution • For this type of RR, the general solution is a linear combination of linearly independent solutions.

26. Solution?

27. Assignment • Read: Recurrence Relations Notes, Part II • HW #5: Part I (Section 1.2) Exercises in the RR notes.

28. End

29. Extra: Big Picture • Look at the big picture: functions as “systems”

30. x y x f y 1 2 2 4 3 8 y y 4 16 x x 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Recurrence Relations So why should we care about recurrence relations? Generalizes the idea of a function to an operator or dynamic system A function is like a table that takes a number in and gives a number out.

31. f y1 x1 y2 x1 x2 y3 x1 x2 x2 Recurrence Relations Functions can also take in a vector of numbers or yield a vector out.

32. f y x k k 1 1 2 2 3 3 4 4 5 5 6 6 7 7 9 9 8 8 Recurrence Relations What if the input and output vectors of a function were infinitely long? Then xand y themselves are functions! f becomes a map from one function to another! We call maps that take functions as inputs or generate functions as outputs operators, or systems. An operator is like a rule, it’s the mathematical description of a subroutine!

33. y x k k 1 1 2 2 3 3 4 4 5 5 6 6 7 7 9 9 8 8 Recurrence Relations y(k) x(k) S

34. Recurrence Relations y(k) x(k) S Unlike a simple function, an operator can have local variables that store intermediate calculations—just like a subroutine. In other words, S is a map that can have memory!

35. y(0)=yo Math Description Recurrence Relations y(k) x(k) S Subroutine Description static memory: w = 0 function S(x,k) if k=0 then y f yo else y f w + x w f y return y

36. y(0)=yo Math Description Subroutine Description static memory: w = 0 function S(x,k) if k=0 then y f yo else y f 5w + x w f y return y Recurrence Relations y(k) x(k) S

37. y(0)=yo, y(1) = y1 Math Description Recurrence Relations What if more than one number is stored in memory? y(k) x(k) S Subroutine Description static memory: w0, w1 = 0 function S(x,k) if k=0 then y f yo else if k=1 then y f y1 else y f a1w1 + aow0+x w0f w1 w1f y return y