Algorithm Analysis

Algorithm Analysis CS 201 Fundamental Structures of Computer Science

Introduction • Once an algorithm is given for a problem and decided to be correct, an important step is to determine how much in the way of resources (such as time or space) the algorithm will require. • In this chapter, we shall discuss • How to estimate the time required for a program

Mathematical background • The idea of the following definitions is to establish a relative order among functions. • Given two functions, there are usually points where one function is smaller than the other function. • It does not make sense to claim f(N) < g(N) • Let us compare f(N) = 1000N and g(N) = N2 • Thus, we will compare their relative rates of growth

Mathematical background • Definition 1:

Examples • f(N)=N3 grows faster than g(N)=N2, sog(N) = O(f(N)) or f(N) = (g(N)) • f(N)=N2 and g(N)=2N2 grow at the same rate, sof(N) = O(g(N)) and f(N) = (g(N)) • If g(N)=2N2, g(N)=O(N4), g(N)=O(N3), g(N)=O(N2) are all technically correct, but the last one is the best answer. • Do not say T(N)=O(2N2) or T(N)=O(N2+N). The correct form is T(N)=O(N2).

Mathematical background • Rule 1: • Rule 2: • Rule 3:

Examples • Determine which of f(N)=N logN and g(N)=N1.5 grows faster.  Determine which of logN and N0.5 grows faster.  Determine which of log2N and N grows faster.  Since N grows faster than any power of a log, g(N) grows faster than f(N).

Examples • Consider the problem of downloading a file over the Internet. • Setting up the connection: 3 seconds • Download speed: 1.5 Kbytes/second • If a file is N kilobytes, the time to download is T(N) = N/1.5 + 3, i.e., T(N) = O(N). • 1500K file takes 1003 seconds • 750K file takes 503 seconds • If the connection speed doubles, both times decrease, but downloading 1500K still takes approximately twice the time downloading 750K.

Typical growth rates

Plots of various algorithms

Algorithm analysis • The most important resource to analyze is generally the running time • We assume that simple instructions (such as addition, comparison, and assignment) take exactly one unit time • Unlike the case with real computers • For example, I/O operations take more time compared to comparison and arithmetic operators • Obviously, we do not have this assumption for fancy operations such as matrix inversion, list insertion, and sorting. • We assume infinite memory • We do not include the time required to read the input

Algorithm analysis • Typically, the size of the input is the main consideration • Worst-case performance represents a guarantee for performance on any possible input • Average-case performance often reflects typical behavior • Best-case performance is often of little interest • Generally, it is focused on the worst-case analysis • It provides a bound for all inputs • Average-case bounds are much more difficult to compute

Algorithm analysis • Although using Big-Theta would be more precise, Big-Oh answers are typically given. • Big-Oh answer is not affected by the programming language • If a program is running much more slowly than the algorithm analysis suggests, there may be an implementation inefficiency • e.g., This can occur in C++ when arrays are inadvertently copied in their entirety instead of passed with references.

Algorithm analysis • If two programs are expected to take similar times, probably the best way to decide which is faster to code them both up and run them! • On the other hand, we would like to eliminate the bad algorithmic ideas early by algorithm analysis • Although different instructions can take different amounts of time (these would correspond to constants in Big-Oh notation), we ignore this difference in our analysis and try to find the upper bound of the algorithm

A simple example • Estimate the running time of the following function int sum(int n){ int partialSum = 0; for (int i = 0; i <= n; i++) partialSum += i * i * i; return partialSum; }

General rules • Rule 1 – for loops: The running time of a for loop is at most the running time of the statements inside the for loop (including tests) times the number of iterations • Rule 2 – nested loops: Analyze these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops for (i = 0; i < n; i++) for (j = i; j < n; j++) k++;

General rules • Rule 3 – consecutive statements: just add. for (i = 0; i < n; i++) a[i] = 0; for (i = 0; i < n; i++) for (j = 0; j < n; j++) a[i] += a[j] + i + j; • Rule 4 – if/else: For the following, the running time is never more than that of the test plus the larger of that of S1 and S2 if (condition) S1 else S2

General rules • If there are function calls, these must be analyzed first • If there are recursive functions, be careful about their analyses. For some recursions, the analysis is trivial long factorial(int n){ if (n <= 1) return 1; return n * factorial(n-1); }

General rules • Let us analyze the following recursion long fib(int n){ if (n <= 1) return 1; return fib(n-1) + fib(n-2); } • By induction, it is possible to show that • So the running time grows exponentially. By using a for loop, the running time can be reduced substantially

Sequential search index = -1; for (i = 0; i < N; i++) if (A[i] == value){ index = i; break; } • Worst-case: It is the last element or it is not found • N iterations  O(N) • Best-case: It is the first element • 1 iteration  O(1) • Average case: We may have N different cases • (N + 3) / 2 iterations  O(N)

Searching a value in a sorted array using binary search int binarySearch(int *A, int N, int value){ int low = 0, high = N; while (low <= high){ int mid = (low + high) / 2; if (a[mid] < value) low = mid + 1; else if (a[mid] > value) high = mid - 1; else return mid; } return -1; }

Iterative solution of raising an integer to a power long pow1(long x, long n){ long result = 1; for (int i = 1; i <= n; i++) result = result * x; return result; }

Recursive solution of raising an integer to a power long pow2(long x, long n){ if (n == 0) return 1; if (n == 1) return x; if (n % 2 == 0) return pow2(x*x,n/2); else return pow2(x*x,n/2) * x; }

Algorithm Analysis