CSE 246 Data Structures and Algorithms

1. CSE 246Data Structures and Algorithms Spring2011 Lecture#9

2. Analyzing Algorithm • In this section, we examine a means of analyzing the performance of an algorithm. • Usually we are interested in the amount of memory required by an algorithm – its space complexity– and the time taken for an algorithm to run – its time complexity. • In this course we will consider only the time complexity of algorithms. Quratulain

3. Introduction to Time Complexity • An important question is: How efficient is an algorithm or piece of code? Efficiency covers lots of resources, including: • CPU (time) usage • memory usage • disk usage • network usage Quratulain

4. Introduction to Time Complexity • Be careful to differentiate between: • Performance: how much time/memory/disk/... is actually used when a program is run. This depends on the machine, compiler, etc. as well as the code. • Complexity: how do the resource requirements of a program or algorithm scale, i.e., what happens as the size of the problem being solved gets larger. • Complexity affects performance but not the other way around. Quratulain

5. Introduction to Time Complexity • To do this exactly, we should count the number of CPU cycles it takes to perform each operation in the algorithm. • Needless to say, this would be a bit tedious and is not a very practical approach. • Instead we will make a simplification. We will count the number of times simple statements are executed. • By simple statement we mean a statement that is executed in an amount of time T. • we are usually interested in the worst case: what are the max operations that might be performed for a given problem size (other cases -- best case and average case) Quratulain

6. Analysis of Algorithms Dilemma: you have two (or more) methods to solve problem, how to choose the BEST? One approach: implement each algorithm in java, test how long each takes to run. Problems: • Different implementations may cause an algorithm to run faster/slower • Some algorithms run faster on some computers • Algorithms may perform differently depending on data (e.g., sorting often depends on what is being sorted) Quratulain

7. Analysis of Algorithms Analysis includes: • Concept of "best" What to measure • Types of "best” best-, average-, worst-case • Comparison methods Big-O analysis • Examples Sequential search and binary search Quratulain

8. Analysis: A Better Approach Idea: characterize performance in terms of key operation(s) • Sorting: • count number of times two values compared • count number of times two values swapped • Searching: • count number of times value being searched for is compared to values in array • Recursive function: • count number of recursive calls Quratulain

9. Analysis in General Want to comment on the “general” performance of the algorithm • Measure for several examples, but what does this tell us in general? Instead, assess performance in an abstract manner Idea: analyze performance as size of problem grows Examples: • Sorting: how many comparisons for array of size N? • Searching: #comparisons for array of size N May be difficult to discover a reasonable formula Quratulain

10. Analysis where Results Vary Types of analyses: • Best-case: what is the fastest an algorithm can run for a problem of size N? • Average-case: on average how fast does an algorithm run for a problem of size N? • Worst-case: what is the longest an algorithm can run for a problem of size N? Quratulain

11. How to Compare Formulas? Answer depends on value of N: N 50N2+31N3+24N+15 3N2+N+21+4*3N 1 120 37 2 511 71 3 1374 159 4 2895 397 5 5260 1073 6 8655 3051 7 13266 8923 8 19279 26465 9 26880 79005 10 36255 236527 Quratulain

12. What Happened? N 1 37 12 32.4 2 71 36 50.7 3 159 108 67.9 4 397 324 81.6 5 1073 972 90.6 6 3051 2916 95.6 7 8923 8748 98.0 8 26465 26244 99.2 9 79005 78732 99.7 10 236527 236196 99.9 • One term dominated the sum Quratulain

13. As N Grows, Some Terms Dominate Quratulain

14. Order of Magnitude Analysis Measure speed with respect to the part of the sum that grows quickest Ordering: Quratulain

15. Order of Magnitude Analysis (cont) Furthermore, simply ignore any constants in front of term and simply report general class of the term: grows proportionally to grows proportionally to When comparing algorithms, determine formulas to count operation(s) of interest, then compare dominant terms of formulas Quratulain

16. Analyzing Running Time T(n), or the running time of a particular algorithm on input of size n, is taken to be the number of times the instructions in the algorithm are executed. Pseudo code algorithm illustrates the calculation of the mean (average) of a set of n numbers: 1. n = read input from user 2. sum = 0 3. i = 0 4. while i < n 5. number = read input from user 6. sum = sum + number 7. i = i + 1 8. mean = sum / n The computing time for this algorithm in terms on input size n is: T(n) = 4n + 5. Statement Number of times executed 1 1 2 1 3 1 4 n+1 5 n 6 n 7 n 8 1 Quratulain

17. Big-O Notation • Computer scientists like to categorize algorithms using Big-O notation. • If you tell a computer scientist that they can choose between two algorithms: • one whose time complexity is O( N ) • Another is O( N2 ), then the O( N ) algorithm will likely be chosen. Let’s find out why… Quratulain

18. Big-O Notation • Let’s consider a function on N, T(N) N is a measure of the size of the problem we try to solve, e.g., Definition: T(N) is O( g(N) ) if: So, if T(N) is O( g(N) ) then T(N) grows no faster than g(N). Example 1 • Suppose, for example, that T is the time taken to run an algorithm to process data in an array of length N. We expect that T will be a function of N and hence write: • If we now say that T(N) is O(N) we are saying that T(N) grows no faster than N. • This is good news – it tells us that if we double the size of the array, we can expect that the time taken to run the algorithm will double (roughly speaking). Quratulain

19. Example Suppose f(n) = n2 + 3n - 1. We want to show that f(n) = O(n2). f(n) = n2 + 3n - 1 < n2 + 3n (subtraction makes things smaller so drop it) <= n2 + 3n2 (since n <= n2 for all integers n) = 4n2 Therefore, if C = 4, we have shown that f(n) = O(n2). Notice that all we are doing is finding a simple function that is an upper bound on the original function. Because of this, we could also say that This would be a much weaker description, but it is still valid. f(n) = O(n3) since (n3) is an upper bound on n2 Quratulain

20. Big-Oh Operations Summation Rule Suppose T1(n) = O(f1(n)) and T2(n) = O(f2(n)). Further, suppose that f2 grows no faster than f1, i.e., f2(n) = O(f1(n)). Then, we can conclude that T1(n) + T2(n) = O(f1(n)). More generally, the summation rule tells us O(f1(n) + f2(n)) = O(max(f1(n), f2(n))). Proof: Suppose that C and C' are constants such that T1(n) <= C * f1(n) and T2(n) <= C' * f2(n). Let D = the larger of C and C'. Then, T1(n) + T2(n) <= C * f1(n) + C' * f2(n) <= D * f1(n) + D * f2(n) <= D * (f1(n) + f2(n)) <= O(f1(n) + f2(n)) Quratulain

21. Big-Oh Operations Product Rule Suppose T1(n) = O(f1(n)) and T2(n) = O(f2(n)). Then, we can conclude that T1(n) * T2(n) = O(f1(n) * f2(n)). The Product Rule can be proven using a similar strategy as the Summation Rule proof. • Analyzing Some Simple Programs (with No Sub-program Calls) • General Rules: • All basic statements (assignments, reads, writes, conditional testing, library calls) run in constant time: O(1). • The time to execute a loop is the sum, over all times around the loop, of the time to execute all the statements in the loop, plus the time to evaluate the condition for termination. Evaluation of basic termination conditions is O(1) in each iteration of the loop. • The complexity of an algorithm is determined by the complexity of the most frequently executed statements. If one set of statements have a running time of O(n3) and the rest are O(n), then the complexity of the algorithm is O(n3). This is a result of the Summation Rule. Quratulain

22. Question • Suppose you have two algorithms (Algorithm A and Algorithm B) and that both algorithms are O(N). Does this mean that they both take the same amount of time to run? Quratulain

23. Example We will now look at various Java code segments and analyze the time complexity of each: int count = 0; int sum = 0; while( count < N ) { sum += count; System.out.println( sum ); count++; } Total number of constant time statements executed...? Quratulain

24. Example int count = 0; int sum = 0; while( count < N ) { int index = 0; while( index < N ) { sum += index * count; System.out.println( sum ); index++; } count++; } Total number of constant time statements executed ….? Quratulain

25. Example int count = 0; while( count < N ) { int index = 0; while( index < count + 1 ) { sum++; index++; } count++; } Total number of constant time statements executed…? Quratulain

26. Example int count = N; int sum = 0; while( count > 0 ) { sum += count; count = count / 2; } Total number of constant time statements executed…? Quratulain