Algorithmic Time Complexity Basics

1. Algorithmic Time Complexity Basics ShantanuDutt ECE Dept. UIC

2. Time Complexity • An algorithm time complexity is a function T(n) of problem size n that represents how much time the algorithm will take to complete its task. • Note that there could be more than one problem size parameter n, in which case we can denote the time complexity function as T(S), where S is the set of size parameters. E.g., for the shortest path problem on a graph G, we have 2 size parameters, n the # of vertices and e the # of edges (thus T(S) = T(n,e)); for the covering part of Quine-McCluskey (QM), we also have 2 size parameters, m the # of MTs and p the # of PIs (T(S) = T(m,p)). • In general, the runtime of an algorithm/program is not only determined by the algorithm, but also the processor speed, memory size and speed, bus speed, etc. However, T(n) generally overlooks technology parameters, and focuses on the intrinsic # of basic steps that an algorithm has to perform as a function of n. • The main job of T(n) is representing how the algorithm’s runtime grows as a function of n, as opposed to giving us an absolute time that the algorithm will take to solve its problem. • Thus if T(n) = n, then runtime doubles as n doubles, if T(n) = n2 then runtime increases by 4 times as n doubles ((2n)2) = 4n2), and if T(n) = 2n, then runtime increases by an exponent of 2 (i.e., quadratically) as n doubles (22n = (2n)2).

3. Time Complexity • T(n) is determined by counting the number, as a function of n, of basic steps that do not depend upon the input size (i.e., are constant-time operations) that the algorithm has to perform. Note that different basic steps (e.g., addition, multiplication of integers) may take different absolute times, but each counts as 1 operation Count # of basic steps that need to be performed by algorithm A that are independent of n (i.e., are constant time operations) n T(n)

4. Time Complexity: Big O Notation • The Big O notation for T(n) specifies the closest or smallest upper bound function f(n) for T(n) (denoted by T(n) = O(f(n)), which essentially says that for a large enough n, T(n) <= f(n). • Formally, we say that for two monotonic function T(n) and f(n), T(n) = O(f(n)), if there exist a constant c, and an n0, s.t. T(n) <= c*f(n), for n >= n0 • The following is a graphical representation of the T(n) = O(f(n)) relation: f(n) T(n) Ack: Graph obtained from http://www.cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html

5. Time Complexity: Big O Notation • Example: T(n) = n2 + 2n + 3, then f(n) = n2, i.e., T(n) = O(n2). Proof: For n >= 2 (= n0), T(n) = n2 + 2n + 3 < n2 + n2 + n2 = 3n2. Thus for n0 = 2 and c = 3, T(n) <= c*n2 for n >= n0 • In general, to determine the big O complexity term for T(n), determine its most dominant term (the term that will be greater than all other terms for a large enough n), get rid of all constants in the term, and if this simplified term is f(n), then T(n) = O(f(n)) • f(n) is also called the asymptotic complexity of T(n) • Sometimes, it is important to retain the multiplicative constants or exponentiation constants, or all constants (e.g., in other complexity measures such as hardware cost). Thus it may be important to not simplify 3n2 to n2. This depends on the detail to which the algorithm is being analyzed and the complexity of a competing algorithm.

6. Time Complexity: Big W, Q Notations • Definition of "big Omega" • We need the notation for the lower bound. A capital omega Ω notation is used in this case. We say that T(n) = Ω(g(n)) when there exists constant c that T(n) ≥ c*g(n) for all sufficiently large n (n >= n0). Examples: • n = Ω(1) • n2 = Ω(n) • n2 = Ω(n log(n)) • 2 n + 1 = Ω (n) • Definition of "big Theta" • To measure the complexity of a particular algorithm, means to find the upper and lower bounds. A new notation is used in this case. We say that T(n) = Θ(g(n)) if and only T(n) = O(g(n)) and T(n) = Ω(g(n)). If a worst-case analysis is quite exact in terms of the dominating term in T(n), we can say T(n) = Θ(g(n)) rather than T(n) = O(g(n)). Examples: • 2 n = Θ(n) • n2 + 2 n + 1 = Θ( n2) Ack: Obtained from http://www.cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html

7. Time Complexity: Big O, Q Notations • Graphs illustrating Big O, Q Notations: T(n) T(n) T(n) T(n) Figure:  The asymptotical bounds O and Θ Ack: Graph obtained from http://www.leda-tutorial.org/en/official/ch02s02s03.html

8. Time Complexity: Big O, QNotations—When we can use Q and when we cannot • Usage O, Q Notations: When our analysis of T(n) (or any counting function) is exact at least for the dominating term, we can and should use Q. Otherwise we use the O notation. • Example 1: • m(n) = worst-case # of MTs of a non-trivial n-variable function f( ) (!= 0 or 1). Certainly m(n) < 2n. If we choose MTs that have an even # of 1’s in their binary representation, we have 2n-1 MTs, and the function is non-trivial (it is an even-parity function). Thus m(n) = O(2n-1) = O(2n). • Q is whether we can say that m(n), in the worst-case, can be at least of the order of 2n-1. Since our analysis above was exact, we can say, yes. More formally, if we choose a constant c2 = 0.5, then the # of MTs in an even parity function > c2*2n-1, and this m(n) > c2*2n-1. Thus we have m(n) = W (2n-1), and thus m(n) = W (2n) (2n and 2n-1 are of the same “order” as they differ only by a multiplicative constant of 2). • From the above, m(n) = Q(2n). • Example 2: • p(n) = worst-case # of PIs of an n-variable function f( ). We know that across all n-variable functions the # of PIs (or the # of ternary notations w/ symbols 0,1,X) are 3n. So we can say, p(n) = O(3n). • However, the above analysis is not exact, in the sense that for this analysis at least, we do not know for certain if there is any function that has p(n) of the order of 3n. Thus, we cannot say that p(n) = W(3n), and this we cannot say that p(n) = Q(3n). • Note that as has been shown earlier, we know that the max. # of PIs covering a MT can be of the order of nCn/2 ~ 2n-1. We can have a function f( ) that is the sum of these PIs. Thus we can say that in the worst-case p(n) = W (2n). But since 3n is not of the same order as 2n (3n = (2**n)(log 3)), i.e., 3n != Q (2n) (3n = W (2n), but 3n != O(2n)), we cannot say that p(n) = W(3n). Ack: Graph obtained from http://www.leda-tutorial.org/en/official/ch02s02s03.html

9. Different Types of Time Complexity Analysis • The term analysis of algorithms is used to describe approaches to the study of the performance of algorithms. The most prevalent type isworst-case runtime complexity of the algorithm is the function defined by the maximum number of steps taken on any instance of size a. • Thebest-case runtime complexity of the algorithm is the function defined by the minimum number of steps taken on any instance of size a. • Theaverage-case runtime complexity of the algorithm is the function defined by an average number of steps taken on any instance of size a. • Theamortized runtime complexity of the algorithm is the function defined by a sequence of operations applied to the input of size a and averaged over time. • Example. Let us consider an algorithm of sequential searching for an element in an array of size n. • Its worst-case runtime complexity is O(n) = Θ(n) • Its best-case runtime complexity is O(1)  • Its average-case runtime complexity is O(n/2) = O(n) = Θ(n) —Why? Ack: Obtained from http://www.cs.cmu.edu/~adamchik/15-121/lectures/Algorithmic%20Complexity/complexity.html

10. Examples of Time Complexity Analysis . This also is = Q(n3). Generally, important only to count “main” operations, here only additions and multiplications: n3 additions and n3 multiplications, and we arrive at the same O and Q notation complexities of n3 Ack: Adapted from http://www.cs.cornell.edu/courses/cs211/2005sp/Lectures/L14-BigO/L14-15-Complexity.4up.pdf

11. Examples of Time Complexity Analysis Ack: Obtained from http://www.cs.cornell.edu/courses/cs211/2005sp/Lectures/L14-BigO/L14-15-Complexity.4up.pdf

12. Merge Sort (MS) Analysis • One way to solve recurrences, is to open it up and obtain a series of terms to be summed. • Below, we obtain the complexity of merge sort directly (ignoring the recurrence). The main basic operation is a comparison of 2 elements/numbers Merge(n): Worst-case comparisons = ? Best-case comparisons = ? MS(n) Legend: : computation break-up arrow : data transfer/dependency arrow Total comparisons @ level 1: worst-case: ? best-case: ? Merge(n) MS(n/2) MS(n/2) Total comparisons @ level 2: worst-case: ? best-case: ? Merge(n/2) Merge(n/2) MS(n/4) MS(n/4) MS(n/4) MS(n/4) Total comparisons @ last merge level (?) : worst-case: ? best-case: ? 1 1 1 1 Total worst-case complexity: ? Total best-case complexity: ?

13. Importance of Asymptotic Analysis—Worst- & Average-Case • Asymptotic analysis tells us whether a technique/algorithm will be practical in all cases (worst-case analysis) or in the average-case (av.-case analysis) for problem sizes of interest Ack: Table obtained from http://www.csd.uwo.ca/courses/CS1037a/notes/topic13_AnalysisOfAlgs.pdf

14. Importance of Asymptotic Analysis—Worst- & Average-Case (contd.) • Assume each basic oper. takes 1 ms T(n); n Table: T(n) values in ms’s Ack: Obtained from http://www.cs.cornell.edu/courses/cs211/2005sp/Lectures/L14-BigO/L14-15-Complexity.4up.pdf

15. Importance of Asymptotic Analysis—Worst- & Average-Case (contd.) T(n); n Ack: Obtained from http://www.cs.cornell.edu/courses/cs211/2005sp/Lectures/L14-BigO/L14-15-Complexity.4up.pdf

16. Asymptotic Complexity and Efficient Algorithms Ack: Obtained from http://www.cs.cornell.edu/courses/cs211/2005sp/Lectures/L14-BigO/L14-15-Complexity.4up.pdf

17. Concluding Remarks Ack: Obtained from http://www.cs.cornell.edu/courses/cs211/2005sp/Lectures/L14-BigO/L14-15-Complexity.4up.pdf