Analysis of Algorithms CS 4413

1. Analysis of AlgorithmsCS 4413 Briana B. Morrison With thanks to Dr. Hung

2. Algorithmics: the study of algorithms. • Algorithmics: is the core of computer science. • Algorithmics: the spirit of computing. • How important is this course?!

3. The Course • Purpose: a rigorous introduction to the design and analysis of algorithms • Not a lab or programming course • Not a math course, either • But, need know how to program! • Minimum math knowledge!

4. Course Objectives • Basic design concepts and analysis methods of algorithms • Design methodologies include, but not limited to: • Searching/Sorting • Divide-and-Conquer • Greedy Method • Dynamic Programming • Search and Traversal • Branch-and-Bound

5. Programming/project grading • Elegant solution • Logical thinking • Quality of writing and coding • Concise writing is a virtue. • Tell the reader only what is important.

6. Research Project Guidelines • Research projects should consist of the following components: • Define the problem to be solved • Background information • Algorithms and Applications • Implementation • Drawbacks • New applications and discoveries

7. Research Project Guidelines • Some examples of research projects: • Dynamic Programming • SNAKE • Marko Chain Monte Carlo (MCMC) (and Image analysis) • Wavelet Transforms • Others

8. From Problems to Programs • There are many steps involved in writing a computer program to solve a given problem. • The steps go from • 1) problem formulation and specification, • 2) to design of the solution, • 3) to implementation, testing and documentation, and finally • 4) toevaluation of the solution.

9. From Problems to Programs … • Algorithms and Data Structures are the building blocks of most computer programs. • Half the battle is knowing what problem to solve. • Almost any branch of mathematics or science can be called into service to help model some problem domain. • Examples: problems essentially numerical in nature.

10. Algorithms An algorithm is a finite sequence of instructions, each of which has a clear meaning and can be performed with a finite amount of effort in a finite length of time. • Pseudo-language is used to present algorithms. • Pseudo-language is a combination of the constructs of a programming language together with informal English statements.

11. Algorithms … • Example 1: A mathematical model can be used to help design a traffic light for a complicated intersection of roads (figure 1.1). Figure 1.1. An intersection.

12. Figure 1.2. Graph showing incompatible turns.

13. Algorithms … Figure 1.3. Table of incompatible turns.

14. Graph Coloring Problem Figure 1.4. A graph.

15. Heuristic • An algorithm that quickly produces good but not necessarily optimal solutions is called a heuristic. • One reasonable heuristics for graph coloring is the so-called “greedy” algorithm.

16. Greedy Algorithm • Initially we try to color as many vertices as possible with the first color, then as many as possible of the uncolored vertices with the second color, and so on. • To color vertices with a new color, we perform the folowing steps: 1) Select some uncolored vertex and color it with the new color. 2) Scan the list of uncolored vertices. For each uncolored vertex, determine whether it has an edge to any vertex already colored with the new color. If there is no such edge, color the present vertex with the new color.

17. Graph Coloring Problem … Figure 1.5. A coloring of the graph of Fig. 1.2.

18. Pseudo-Language and Stepwise Refinement • Once we have an appropriate mathematical model for a problem, we can formulate an algorithm in terms of that model. • The initial versions of the algorithm are often couched in general statements that will have to be refined subsequently into smaller, more definite instructions.

19. First refinement of greedy algorithm procedure greedy ( var G: GRAPH; Var newclr: SET ); { greedy assigns to newclr a set of vertices of G that may be given the same color} . begin newclr := 0; for each uncolored vertex v of G do if v is not adjacent to any vertex in newclr then begin mark v colored; add v to newclr end end; {greedy}

20. Refinement of greedy algorithm procedure greedy ( var G: GRAPH; var newclr: SET ); begin newclr := 0; for each uncolored vertex v of G do begin found := false; for each vertex w in newclr do if there is an edge between v and w in G then found := true; if found = false then begin { v is adjacent to no vertex in newclr } mark v colored; add v to newclr end end end; {greedy}

21. Refined greedy algorithm procedure greedy ( var G: GRAPH; var newclr: LIST); { greedy assigns to newclr those vertices that may be given the same color} var found: boolean; v, w: integer; begin newclr := 0; v : = first uncolored vertex in G; while v < > null do begin found: = false; w : = first vertex in newclr; while w < > null do begin if there is an edge between v and w in G then found := true; w : = next vertex in newclr end; if found = false do begin mark v colored; add v to newclr end; v : = next uncolored vertex in G end end; {greedy}

22. Abstract Data Types • Procedures generalize the notion of an operator. • Another advantage of procedures is that they can be used to encapsulate parts of an algorithm by localizing in one section of a program all the statements relevant to a certain aspect of a program.

23. Abstract Data Types (ADT) … • Think of ADT as a mathematical model with a collection of operations defined on that model. • Example: Set of integers, together with the operations of union, intersection, and set difference, form a simple example of an ADT. • The two properties of procedures mentioned above – generalization and encapsulation – apply equally well to abstract data types. • An implementation of an ADT is a translation, into statements of a programming language.

24. Data types, Data Structures, and Abstract Data Types • In a programming language, the data type of a variable is the set of values that the variable may assume. • Abstract Data Type is a mathematical model, together with various operations defined on the model. • To represent the mathematical model underlying an ADT we use data structures, which are collection of variables, possibly of several different data types, connected in various ways.

25. An The problem solving process program

26. An Introduction Introduction Proof by induction Asymptotic notation

27. Review: Induction • Suppose • S(k) is true for fixed constant k • Often k = 0 • S(n)  S(n+1) for all n >= k • Then S(n) is true for all n >= k

28. Proof By Induction • Claim:S(n) is true for all n >= k • Basis: • Show formula is true when n = k • Inductive hypothesis: • Assume formula is true for an arbitrary n • Step: • Show that formula is then true for n+1

29. Induction Example: Gaussian Closed Form • Prove 1 + 2 + 3 + … + n = n(n+1) / 2 • Basis: • If n = 0, then 0 = 0(0+1) / 2 • Inductive hypothesis: • Assume 1 + 2 + 3 + … + n = n(n+1) / 2 • Step (show true for n+1): 1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1) = n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2 = (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2

30. Induction Example:Geometric Closed Form • Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a  1 • Basis: show that a0 = (a0+1 - 1)/(a - 1) a0 = 1 = (a1 - 1)/(a - 1) • Inductive hypothesis: • Assume a0 + a1 + … + an = (an+1 - 1)/(a - 1) • Step (show true for n+1): a0 + a1 + … + an+1 = a0 + a1 + … + an + an+1 = (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)

31. Induction Example: Gaussian Closed Form • Prove that 12 + 22 + 32 + … + n2 = (2n3 + 3n2 + n)/6 (sum of squares) • What is this called? • Polynomial Series.

32. Induction… • We’ve been using weak induction • Strong induction also holds • Basis: show S(0) • Hypothesis: assume S(k) holds for arbitrary k <= n • Step: Show S(n+1) follows • Another variation: • Basis: show S(0), S(1) • Hypothesis: assume S(n) and S(n+1) are true • Step: show S(n+2) follows

33. Chapter 1 Algorithms: Efficiency, Analysis, and Order

34. Algorithms • Looking at individual modules that accomplish a specific task. • Need an algorithm that will solve all instances (with any values of parameters) of problem • Solution needs to be efficient – solves problem within the required resource constraints

35. Algorithms • Problem: is a questions to which we seek an answer. • Example: Sort a list S of n numbers in nondecreasing order. The answer is the numbers in sorted sequence. • Other examples?

36. Algorithms … • 1.1 - Sequential search • 1.2 - Add array members • 1.3 - Exchange sort (bubble sort) • 1.4 - Matrix multiplication • 1.5 – Binary search

37. Developing efficient Algorithms • Sequential search versus binary search • 1.6 - The Fibonacci sequence (recursive) • 1.7 - The Fibonacci sequence (iterative)

38. Analysis of Algorithms • Complexity analysis • Analysis of algorithm 1.2 • Analysis of algorithm 1.3 • Analysis of algorithm 1.4 • Analysis of algorithm 1.5 • Analysis of algorithm 1.1

39. Analysis of Algorithms • Algorithm analysis measured the efficiency of an algorithm as the input size becomes large. • Critical resource is most often its running time • Can also take into account the space required to run the program

40. Complexity Analysis • Looking for a measure that is independent of the computer, the programming language, the programmer, and all the complex details of the algorithm. • Analyze the algorithm efficiency by determining the number of times some basic operation is done as a function of the size of the input.

41. Complexity Analysis… • After determining the input size, pick some instruction (or group of instructions) such that the total work done by the algorithm is roughly proportional to the number of times this instruction is done. • In general, time complexity analysis of an algorithm is the determination of how many times the basic operation is done for each value of the input size.

42. Order • What is order? • An intuitive introduction to order • A rigorous introduction to order • Using a limit to determine order

43. Order … • Is 0.01n2 > 100n? • Linear-time algorithms • Quadratic-time algorithms ( algorithm) • Pure quadratic • Complete quadratic

44. Asymptotic Performance • In this course, we care most about asymptotic performance • How does the algorithm behave as the problem size gets very large? • Running time • Memory/storage requirements • Bandwidth/power requirements/logic gates/etc.

45. Asymptotic Notation • By now you should have an intuitive feel for asymptotic (big-O) notation: • What does O(n) running time mean? O(n2)?O(n log n)? • How does asymptotic running time relate to asymptotic memory usage? • Our first task is to define this notation more formally and completely

46. Analysis of Algorithms • Analysis is performed with respect to a computational model • We will usually use a generic uniprocessor random-access machine (RAM) • All memory equally expensive to access • No concurrent operations • All reasonable instructions take unit time • Except, of course, function calls • Constant word size • Unless we are explicitly manipulating bits

47. Input Size • Time and space complexity • This is generally a function of the input size • E.g., sorting, multiplication • How we characterize input size depends: • Sorting: number of input items • Multiplication: total number of bits • Graph algorithms: number of nodes & edges • Etc

48. Running Time • Algorithm FindMax Int findMax (E, n) 1. max = E [0]; 2. for (index = 1; index < n; index ++) 3. if (max < E [index]) 4. max = E [index]; 5. return max;

49. Running Time • Number of primitive steps that are executed • Except for time of executing a function call, most statements roughly require the same amount of time • y = m * x + b • c = 5 / 9 * (t - 32 ) • z = f(x) + g(y) • We can be more exact if need be

50. Analysis • Worst case • Provides an upper bound on running time • An absolute guarantee • Average case • Provides the expected running time • Very useful, but treat with care: what is “average”? • Random (equally likely) inputs • Real-life inputs