Algorithms and Complexity

Algorithms and Complexity

Agenda • Section 1.8: Growth of Functions • Big-O • Big- (Omega) • Big- (Theta) • Section 2.1: Algorithms • Defining Properties • Pseudocode • Section 2.2: Complexity of Algorithms

Section 1.8Big-O, Big-, Big- Big-O notation is a way of comparing functions. Useful for computing algorithmic complexity, i.e. the amount of time that it takes for computer program to run.

Notational Issues The notation is unconventional. Seemingly, an equation is established; however, this is not the intention. EG: 3x 3 + 5x 2 – 9 = O (x 3) Doesn’t mean that there’s a function O (x 3) and that it equals 3x 3 + 5x 2 – 9. Rather the example is read as: “3x 3+5x 2 –9 is big-Oh of x 3” Which actually means: “3x 3+5x 2 –9 is asymptotically dominated by x 3”

Intuitive Notion of Big-O Asymptotic notation captures the behavior of functions for large values of x. EG: Dominant term of 3x 3+5x 2 –9 is x 3. For small x not clear why x 3 dominates more than x 2 or even x;however, as x becomes larger and larger, other terms become insignificant, and only x 3 remains in the picture:

Intuitive Notion of Big-Odomain – [0,2] y = 3x 3+5x 2 –9 y = x 3 y = x 2 y = x

Intuitive Notion of Big-O In fact, 3x 3+5x 2 –9 is smaller than 5x 3 for large enough values of x: y = 5x 3 y = 3x 3+5x 2 –9 y = x 2 y = x

Big-O. Formal Definition The intuition motivates the idea that a function f (x ) is asymptotically dominated by g (x ) if some constant multiple of g (x ) is actually bigger than f (x ) for all large x. Formally: DEF: Let f and g be functions with domain R0 or N and codomain R. If there are constants C and k such  x > k,|f (x )|  C  |g (x )| i.e., past k,f is less than or equal to a multiple of g, then we write: f (x ) = O ( g (x ) )

Common Misunderstanding It’s true that 3x 3 + 5x 2 – 9 = O (x 3) as we’ll prove shortly. However, also true are: • 3x 3 + 5x 2 – 9 = O (x 4) • x 3 = O (3x 3 + 5x 2 – 9) • sin(x)= O (x 4) NOTE: In CS, use of big-O typically involves mentioning only the most dominant term. “The running time is O (x 2.5)” Mathematically big-O is more subtle. It’s a way of comparing arbitrary pairs of functions.

Big-O. Example EG: Show that 3x 3 + 5x 2 – 9 = O (x 3). From the previous graphs it makes sense to let C = 5. Let’s find k so that 3x 3 + 5x 2 – 9  5x 3 for x > k :

EG: Show that3x 3 + 5x 2 – 9 = O (x 3). From the previous graphs it makes sense to let C = 5. Let’s find k so that 3x 3 + 5x 2 – 9  5x 3 for x > k : • Collect terms: 5x 2 ≤ 2x 3 + 9

EG: Show that3x 3 + 5x 2 – 9 = O (x 3). From the previous graphs it makes sense to let C = 5. Let’s find k so that 3x 3 + 5x 2 – 9  5x 3 for x > k : • Collect terms: 5x 2 ≤ 2x 3 + 9 • What k will make 5x 2 ≤ x 3 past k ?

EG: Show that3x 3 + 5x 2 – 9 = O (x 3). From the previous graphs it makes sense to let C = 5. Let’s find k so that 3x 3 + 5x 2 – 9  5x 3 for x > k : • Collect terms: 5x 2 ≤ 2x 3 + 9 • What k will make 5x 2 ≤ x 3 past k ? • k = 5 !

EG: Show that3x 3 + 5x 2 – 9 = O (x 3). From the previous graphs it makes sense to let C = 5. Let’s find k so that 3x 3 + 5x 2 – 9  5x 3 for x > k : • Collect terms: 5x 2 ≤ 2x 3 + 9 • What k will make 5x 2 ≤ x 3 past k ? • k = 5 ! • So for x > 5, 5x 2 ≤ x 3 ≤ 2x 3 + 9

EG: Show that3x 3 + 5x 2 – 9 = O (x 3). From the previous graphs it makes sense to let C = 5. Let’s find k so that 3x 3 + 5x 2 – 9  5x 3 for x > k : • Collect terms: 5x 2 ≤ 2x 3 + 9 • What k will make 5x 2 ≤ x 3 past k ? • k = 5 ! • So for x > 5, 5x 2 ≤ x 3 ≤ 2x 3 + 9 • Solution: C = 5, k = 5 (not unique!)

Big-O. Negative Example x 4O (3x 3 + 5x 2 – 9) : Show that no C, k can exist such that past k, C (3x3 + 5x 2 – 9)x 4 is always true. Easiest way is with limits (yes Calculus is good to know):

Big-O. Negative Example x 4O (3x 3 + 5x 2 – 9) : Show that no C, k can exist such that past k, C (3x3 + 5x 2 – 9)x 4 is always true. Easiest way is with limits (yes Calculus is good to know): Thus no-matter C, x 4 will always catch up and eclipse C (3x3 + 5x 2 – 9)

Big-O and limits Knowing how to use limits can help to prove big-O relationships: LEMMA: If the limit as x   of the quotient |f (x) / g (x)| exists (so is non-infinite) then f (x ) = O ( g (x ) ). EG: 3x 3 + 5x 2 – 9 = O (x 3 ). Compute: …so big-O relationship proved.

Big- and Big- Big- is just the reverse of big-O. I.e. f (x ) = (g (x ))  g (x ) = O (f (x )) So big- says that asymptotically f (x ) dominatesg (x ). Big- says that both functions dominate each-other so are asymptotically equivalent. I.e. f (x ) = (g (x ))  f (x ) = O (g (x ))  f (x ) = (g (x )) Synonym for f = (g): “f is of orderg ”

Useful facts • Any polynomial is big- of its largest term • EG: x 4/100000 + 3x 3 + 5x 2 – 9 =(x 4) • The sum of two functions is big-O of the biggest • EG: x 4 ln(x ) + x 5 = O (x 5) • Non-zero constants are irrelevant: • EG: 17x 4 ln(x ) = O (x 4 ln(x ))

Big-O, Big-, Big-. Examples Q: Order the following from smallest to largest asymptotically. Group together all functions which are big- of each other:

Big-O, Big-, Big-. Examples A: 1. 2. 3. , (change of base formula) 4. 5. 6. 7. 8. 9. 10.

Incomparable Functions Given two functions f (x ) and g (x ) it is not always the case that one dominates the other so that f and g are asymptotically incomparable. E.G: f (x) = |x 2 sin(x)| vs. g (x) = 5x 1.5

Incomparable Functions y = x2 y = |x 2 sin(x)| y = 5x1.5

Incomparable Functions y = x2 y = 5x1.5 y = |x 2 sin(x)|

Example for Section 1.8 Link to example proving big-Omega of a sum.

Section 2.1Algorithms and Pseudocode By now, most of you are well adept at understanding, analyzing and creating algorithms. This is what you did in your (pre-requisite) 1 semester of programming! DEF: An algorithm is a finite set of precise instructions for performing a computation or solving a problem. Synonyms for a algorithm are: program, recipe, procedure, and many others.

Hallmarks of Algorithms The text book lays out 7 properties that an algorithm should have to satisfy the notion “precise instructions” in such a way that no ambiguity arises: • Input. Spell out what the algorithm eats • Output. Spell out what sort of stuff it spits out • Determinism (or Definiteness) At each point in computation, should be able to tell exactly what happens next

Hallmarks of Algorithms • Correctness. Algorithm should do what it claims to be doing. • Finiteness. Finite number of steps for computation no matter what input. • Effectiveness.Each step itself should be doable in a finite amount of time. • Generality. Algorithm should be valid on all possible inputs. Q: Which of the conditions guarantee that an algorithm has no infinite loops?

Hallmarks of Algorithms A: Finiteness & Effectiveness

Pseudocode Starting out, students often find it confusing to write pseudocode, but find it easy to read pseudocode. Sometimes, students are so anti-pseudocode that they’d rather write compiling code in homework and exams. This is a major waste of time! I strongly suggest you use pseudocode. You don’t have to learn the pseudo-Pascal in Appendix 2 of the text-book. A possible alternative: pseudo-Java…

Pseudo-Java Start by representing an algorithm as a Java method. It’s clear that Java specific modifiers such as final, static,etc. are not worth worrying about. So consider the method: int f(int[] a){ int x = a[0]; for(int i=1; i<a.length; i++){ if(x > a[i]) x = a[i]; } return x; } Q: What does this algorithm do?

Pseudo-Java A: It finds the minimum of a sequence of numbers. int f(int[] a){ int x = a[0]; for(int i=1; i<a.length; i++){ if(x > a[i]) x = a[i]; } return x; } Q: Is there an input that causes Java to throw an exception?

Pseudo-Java A: Yes, arrays of length 0. This illustrates one of the dangers of relying on real code. Forces you to worry about language specific technicalities, exceptions, etc.

Pseudo-Java int f(int[] a){ int x = a[0]; for(int i=1; i<a.length; i++){ if(x > a[i]) x = a[i]; } return x; } Q: Are there any statements that are too long winded, or confusing to non-Java people?

Pseudo-Java A: Yes, several: 1. int f(int[] a) • Non-Java people may not realize that this says that input should be an array of int’s • FIX (also helps with some problems below) spell out exactly how the array looks and use subscripts, which also tells us the size of the array: “integer f( integer_array (a1, a2, …, an) )” • Type declaration notation “integer f()” is a succinct way of specifying the output .

Pseudo-Java 2. a[0] • brackets annoying in index evaluation. • Why are we starting at 0? • FIX: Use subscripts: “ai” and start with index 1.

Psuedo-Java 3. for(int i=1; i<a.length; i++) • Counter in for-loop is obviously an int • Change “int i=1” to “i=1” • Really annoying if constantly have to refer to array length using the period operator • Subscripts fixed this. Now “a.length” becomes “n ” • Can’t we just directly iterate over whole sequence and not worry about “i++” (that way don’t have to remember peculiar for-loop syntax) • Change whole statement to “for(i=1 to n)” Q: Were there any unintended limitations that using Java created?

Pseudo-Java A: Yes. We restricted the input/output to be of type int. Probably wanted algorithm to work on arbitrary integers, not just those that can be encapsulated in 32 bits! Java solution: use class java.math.BigInteger…NOT! Way too gnarly! Just output “integer” instead of int. Resulting pseudo-code (also removed semicolons):

Pseudo-Java integer f(integer_array (a1, a2, …, an) ){ x = a1 for(i =2 to n){ if(x > ai) x = ai } return x }

Ill-defined Algorithms. Examples. Q: What’s wrong with each example below? • integer power(a, b) {…rest is okay…} • integer power(integer a, rational b) {…rest is okay…} • boolean f(real x){ // “real” = real no. for(each digit d in x){ if(d == 0) return true; } return false; }

Ill-defined Algorithms. Examples. A: • Input ill-defined. • Must be incorrect as output of fractional power such as 2½ is usually not an integer. • Number of steps infinite on certain inputs. EG: 2.121212121212121212…

Ill-defined Algorithms. Examples. Q: What’s wrong with: String f(integer a) { if(a == 0){ return “Zero” OR return “The cardinality of {}” } }

Ill-defined Algorithms. Examples. A: Two problems • Non-deterministic –what should we output when reading 0? • Non-general –what happens to the input 1?

Algorithm for Surjectivity Give an algorithm that determines whether a function from a finite set to another is onto. Hint : Assume the form of the sets is as simple as possible, e.g. sets of numbers starting from 1. Algorithm doesn’t depend on type of elements since could always use a “look-up table” to convert between one form and another. This assumption also creates cleaner, clearer pseudocode.

Algorithms and Complexity