Data Structure Algorithm Analysis

Data StructureAlgorithm Analysis TA: Abbas Sarraf ab_sarraf@aut.ac.ir

Objectives • How to predict an algorithm’s performance • How well an algorithm scales up • How to compare different algorithms for a problem

Asymptotic Notations • Big-O, “bounded above by”: T(n) = O(f(n)) • For some c and N, T(n) c·f(n) whenever n > N. • Big-Omega, “bounded below by”: T(n) = W(f(n)) • For some c>0 and N, T(n) c·f(n) whenever n > N. • Same as f(n) = O(T(n)). • Big-Theta, “bounded above and below”: T(n) = Q(f(n)) • T(n) = O(f(n)) and also T(n) = W(f(n))

N 0 By Pictures • Big-Oh (most commonly used) • bounded above • Big-Omega • bounded below • Big-Theta • exactly

Examples • What is the big-O of 2 n2 + 1000n + 5 ? • T(n) = 2 n2 + 1000n + 5; f(n) = n2 • T(n) <= c f(n) ? 2 n2 + 1000n + 5 <= c n2 ? • for c = 3, n0 = 1001,check: 2(10012) + 1000(1001) + 5 = 3005007;3(10012) = 3006003 • answer: O(n2) • An algorithm actually takes n5 + n3 + 7, we say its complexity is O(n5). ( proof in the book, p. 54 ) • This is only for addition! Obviously if the algorithm takes n5 * n3 its complexity is O(n8)!

Cost sum = 0; ---> 1 sum = sum + next; ---> 1 Total Cost: 2 Cost for( int i = 1; i <= n; i++ ) ---> 1 + n+1 + n = 2n+2 sum = sum++; ---> n Total Cost: 3n + 2 Cost k = 0 ---> 1 for( int i = 0; i < n; i++ ) ---> 2n+2 for( int j = 0; j < n; j++ ) ---> n(2n+2) = 2n2 +2n k++; ---> n2 Total Cost: 3n2 + 4n + 3

Both algorithms are O(n).

int maxSum = 0; for( int i = 0; i < a.size( ); i++ ) for( int j = i; j < a.size( ); j++ ) { int thisSum = 0; for( int k = i; k <= j; k++ ) thisSum += a[ k ]; if( thisSum > maxSum ) maxSum = thisSum; } return maxSum; Example • Time complexity: O(n3)

2n n2 2n n2 n3 n log n n n3 n log n log n n log n

Class O(1) • Function/order • Constant time • Examples • Find the ith element in an array. • A[i] • Remarks • The running time of the algorithm doesn't depend on the value of n.

Class O(logan) • Function/order • Logarithmic time • Examples • binary search • Remarks • Typically achieved by dividing the problem into smaller segments and only looking at one input element in each segment. • Binary Search: Every time you go through the recursion (binary search uses recursion), the problem is reduced in half. So the most amount of times you can go through the recursion is log2n.

Class O(n) • Function/order • Linear time • Examples • Find the minimum element, printing, listing • Remarks • Typically achieved by examining each element in the input once. • The running time of the algorithm is proportional to the value of n.

Class O(n * log(n)) • Examples • heapsort, mergesort • Remarks • Typically achieved by dividing the problem into subproblems, solving the subproblems independently, and then combining the results. Unlike the log N algorithms, each element in the subproblems must be examined.

Class O(n2) • Function/order • Quadratic time • Examples • bubblesort, insertion sort • Remarks • Typically achieved by examining all pairs of data elements • Comparisons :(1 + 2 + ... + n) = n(n + 1)/2 = n2/2 + n/2 = O(n2)

Example -> what if “if”?! sum = 0; for (i = 0; i < n; i++) { if (is_even(i)) { for (j = 0; j < n; j++) sum++; } else sum = sum + n; } O(n2) : outer loop is O(n) inside the loop: if “true” clause executed for half the values of n -> O(n),if “false” clause executed for other half -> O(1);the innermost loop is O(n),so the complexity is n(n + 1) = O(n2)

Average, Best, and Worst-Case Insertion Sort:

we let tjbe the number of times the while loop test in line 5 is executed for that value of j.

Insertion Sort Best Case: ( tj = 1 ) T(n) = c1n + c2(n - 1) + c4(n - 1) + c5(n - 1) + c8(n - 1) = (c1 + c2 + c4 + c5 + c8)n - (c2+ c4 + c5 + c8). = O(n)

Insertion Sort • Worst Case? • tj = i • Average? (Probabilistic ) • E{ tj } = i / 2 (how?)

Average, Best, and Worst-Case • On which input instances should the algorithm’s performance be judged? • Average case: • Real world distributions difficult to predict • Best case: • Seems unrealistic • Worst case: • Gives an absolute guarantee • We will use the worst-case measure.

int maxSum = 0; for( int i = 0; i < a.size( ); i++ ) for( int j = i; j < a.size( ); j++ ) { int thisSum = 0; for( int k = i; k <= j; k++ ) thisSum += a[ k ]; if( thisSum > maxSum ) maxSum = thisSum; } return maxSum; Example: Given A1,…,An , find the maximum value of Ai+Ai+1+···+Aj0 if the max value is negative • Time complexity: O(n3)

Algorithm 2 into maxSum = 0; for( int i = 0; i < a.size( ); i++ ) int thisSum = 0; for( int j = i; j < a.size( ); j++ ) { thisSum += a[ j ]; if( thisSum > maxSum ) maxSum = thisSum; } return maxSum; • Idea: Given sum from i to j-1, we can compute the sum from i to j in constant time. • This eliminates one nested loop, and reduces the running time to O(n2).

Algorithm 3 2, 3, -2, 1, -5, 4, 1, -3, 4, -1, 2 • Time complexity clearly O(n) • But why does it work? I.e. proof of correctness. int maxSum = 0, thisSum = 0; for( int j = 0; j < a.size( ); j++ ) { thisSum += a[ j ]; if ( thisSum > maxSum ) maxSum = thisSum; else if ( thisSum < 0 ) thisSum = 0; } return maxSum;

Proof of Correctness • Max subsequence cannot start or end at a negative Ai. • More generally, the max subsequence cannot have a prefix with a negative sum. Ex: -2 11 -4 13 -5 -2 • Thus, if we ever find that Ai through Aj sums to < 0, then we can advance i to j+1 • Proof. Suppose j is the first index after i when the sum becomes < 0 • The max subsequence cannot start at any p between i and j. Because Ai through Ap-1 is positive, so starting at i would have been even better.

Algorithm 3 int maxSum = 0, thisSum = 0; for( int j = 0; j < a.size( ); j++ ) { thisSum += a[ j ]; if ( thisSum > maxSum ) maxSum = thisSum; else if ( thisSum < 0 ) thisSum = 0; } return maxSum • The algorithm resets whenever prefix is < 0. Otherwise, it forms new sums and updates maxSum in one pass

Data Structure Algorithm Analysis