Big Oh Notation

1. Big Oh Notation • Greek letter Omicron (Ο) is used to denote the limit of asymptotic growth of an algorithm • If algorithm processing time grows linearly with the input set n, then we say the algorithm is Order n, or O(n). • This notation isolates an algorithm’s run-time from other factors: • Size of the problem set • Initialization time • Processor speed and instruction set

2. Big-Oh notation • Let b(x) be the bubble sort algorithm • We say b(x) is O(n2) • This is read as “b(x) is big-oh n2” • This means that the input size increases, the running time of the bubble sort will increase proportional to the square of the input size • In other words, by some constant times n2 • Let l(x) be the linear (or sequential) search algorithm • We say l(x) is O(n) • Meaning the running time of the linear search increases directly proportional to the input size

3. Big-Oh notation • Consider: b(x) is O(n2) • That means that b(x)’s running time is less than (or equal to) some constant times n2 • Consider: l(x) is O(n) • That means that l(x)’s running time is less than (or equal to) some constant times n

4. Big-Oh proofs • Show that f(x) = x2 + 2x + 1 is O(x2) • In other words, show that x2 + 2x + 1 ≤c*x2 • Where c is some constant • For input size greater than some x • We know that 2x2≥ 2x whenever x ≥ 1 • And we know that x2≥1 whenever x ≥ 1 • So we replace 2x+1 with 3x2 • We then end up with x2 + 3x2 = 4x2 • This yields 4x2≤c*x2 • This, for input sizes 1 or greater, when the constant is 4 or greater, f(x) is O(x2) • We could have chosen values for c and x that were different

5. Big-Oh proofs

6. Sample Big-Oh problems • Show that f(x) = x2 + 1000 is O(x2) • In other words, show that x2 + 1000 ≤ c*x2 • We know that x2 > 1000 whenever x > 31 • Thus, we replace 1000 with x2 • This yields 2x2≤ c*x2 • Thus, f(x) is O(x2) for all x > 31 when c ≥ 2

7. Sample Big-Oh problems • Show that f(x) = 3x+7 is O(x) • In other words, show that 3x+7 ≤ c*x • We know that x > 7 whenever x > 7 • Uh huh…. • So we replace 7 with x • This yields 4x≤c*x • Thus, f(x) is O(x) for all x > 7 when c ≥ 4

8. A variant of the last question • Show that f(x) = 3x+7 is O(x2) • In other words, show that 3x+7 ≤ c*x2 • We know that x > 7 whenever x > 7 • Uh huh…. • So we replace 7 with x • This yields 4x < c*x2 • This will also be true for x > 7 when c≥ 1 • Thus, f(x) is O(x2) for all x > 7 when c ≥ 1

9. What that means • If a function is O(x) • Then it is also O(x2) • And it is also O(x3) • Meaning a O(x) function will grow at a slower or equal to the rate x, x2, x3, etc.

10. Function growth rates • For input size n = 1000 • O(1) 1 • O(log n) ≈10 • O(n) 103 • O(n log n) ≈104 • O(n2) 106 • O(n3) 109 • O(n4) 1012 • O(nc) 103*cc is a consant • 2n ≈10301 • n! ≈102568 • nn 103000 Many interesting problems fall into these categories

11. Function growth rates Logarithmic scale!

12. Integer factorization • Factoring a composite number into it’s component primes is O(2n) • Where n is the number of bits in the number • This, if we choose 2048 bit numbers (as in RSA keys), it takes 22048 steps • That’s about 10617 steps!

13. Formal Big-Oh definition • Let f and g be functions. We say that f(x) is O(g(x)) if there are constants c and k such that |f(x)| ≤ C |g(x)| whenever x > k

14. Formal Big-Oh definition

15. Big Omega (Ω) and Big Theta (Θ) • If Big-Oh a less-than relationship: • then Big Omega is greater-than • | f(x) | >C | g(x) | when x > n0 • and Big Theta is equals • if f(x) is O(g(x)) and Ω(g(x)), then it is Θ(g(x)) • x2is Θ(x2) • x is O(x2) • x2is Ω(x)

16. A useful recursive algorithm • Merge sort • proceduremergesort (L = a1,…an) • if n>1 then • m := floor(n/2) • L1 := a1,a2,…,am • L2 := am+1,am+2,…,an • L := merge(mergesort(L1),mergesort(L2)) • {L is now sorted into elements of increasing order}

17. mergesort needs merge • proceduremerge(L1, L2: lists) • L := empty list • while L1 and L2 are both nonempty • begin • remove smaller of first element of L1 and L2 from the list it is in and put it at the left end of L • if removal of this element make one list empty • then • remove all elements from the other list and append them to L • end • returnL • {L is the merged list with elements in increasing order}

18. Time complexity • First: how many recursive calls are there for n inputs?

19. Satisfiability • Consider a Boolean expression of the form: • (x1 x2 x3)  (x2 x3 x4)  (x1 x4 x5) • This is a conjunction of disjunctions • Is such an equation satisfiable? • In other words, can you assign truth values to all the xi’s such that the equation is true? • The above problem is easy (only 3 clauses of 3 variables) – set x1, x2, and x4 to true • There are other possibilities: set x1, x2, and x5 to true, etc. • But consider an expression with 1000 variables and thousands of clauses

20. Satisfiability • If given a solution, it is easy to check if such a solution works • Plug in the values – this can be done quickly, even by hand • However, there is no known efficient way to find such a solution • The only definitive way to do so is to try all possible values for the n Boolean variables • That means this is O(2n)! • Thus it is not a polynomial time function • NP stands for “Not Polynomial” • Cook’s theorem (1971) states that SAT is NP-complete • There still may be an efficient way to solve it, though!

21. NP Completeness • There are hundreds of NP complete problems • It has been shown that if you can solve one of them efficiently, then you can solve them all • Example: the traveling salesman problem • Given a number of cities and the costs of traveling from any city to any other city, what is the cheapest round-trip route that visits each city once and then returns to the starting city? • Not all algorithms that are O(2n) are NP complete • In particular, integer factorization (also O(2n)) is not thought to be NP complete

22. NP Completeness • It is “widely believed” that there is no efficient solution to NP complete problems • In other words, everybody has that belief • If you could solve an NP complete problem in polynomial time, you would be showing that P = NP • And you’d get a million dollar prize (and lots of fame!) • If this were possible, it would be like proving that Newton’s or Einstein’s laws of physics were wrong • In summary: • NP complete problems are very difficult to solve, but easy to check the solutions of • It is believed that there is no efficient way to solve them

23. Reserve

24. An aside: inequalities • If you have a inequality you need to show: x < y • You can replace the lesser side with something greater: x+1 < y • If you can still show this to be true, then the original inequality is true • Consider showing that 15 < 20 • You can replace 15 with 16, and then show that 16 < 20. Because 15 < 16, and 16 < 20, then 15 < 20

25. An aside: inequalities • If you have a inequality you need to show: x < y • You can replace the greater side with something lesser: x < y-1 • If you can still show this to be true, then the original inequality is true • Consider showing that 15 < 20 • You can replace 20 with 19, and then show that 15 < 19. Because 15 < 19, and 19 < 20, then 15 < 20

26. An aside: inequalities • What if you do such a replacement and can’t show anything? • Then you can’t say anything about the original inequality • Consider showing that 15 < 20 • You can replace 20 with 10 • But you can’t show that 15 < 10 • So you can’t say anything one way or the other about the original inequality