The Fundamentals: Algorithms, Integers, & Matrices

1. The Fundamentals: Algorithms, Integers, & Matrices Chapter 2

2. 2.2 Growth of Functions

3. Big-O notation • A tool to analyze a program's efficiency. • An approximation of • work an algorithm performs • as a function of size of input Work = O(input size)

4. Work • A measure of • effort expended by computer • in performing a computation

5. Commonly used Big-O values (orders of magnitude) Big-O Name O(1) Constant time Lower O(log n) Logarithmic time O(n) Linear time O(n log n) O(n2) Quadratic time O(n3) Cubic time O(nk) Polynomial time O(2n) Exponential time Higher O(n!) Factorial time

6. Table of Common Running Times

7. Growth Rate of Some Functions

8. Big-O • f(x) is O(g(x)) if there are constants C and k such that | f(x) | £ C | g(x) | whenever x > k

9. Show f(x) = x2+2x+1 is O(x2) • Prove: • |x2+2x+1|£ C | x2 | • If we let k > 0, f(x) & g(x) are always positive, we can drop | | • Let C be the sum of f(x) coefficients • x2+2x+1£ 1x2+2x2+1x2 = 4x2 • Note: x2 £ 1x2 and 2x £2x2 and 1£1x2 • This inequality holds beginning at x=1: 4£4 • Choosing k=1, C=4, x2+2x+1 is O(x2)

10. Show f(x) = 7x3 is O(x2) • Prove: • |7x3|£ C | x2 | • Assuming k > 0, divide both sides by x2 • 7x£ C • You choose a constant > any y on y=7x line • No such C exists, since x is arbitrarily large • (Choose any C, and there is an x > C) • \ 7x3 is not O(x2)

11. Big-O Example algorithm • O(1) Assigning value to ith array element; always the same number of steps not necessarily short a program which takes 320 steps, reguardless of input values • O(log n) Binary search, What power of 2 is greater than a input number? A loop whose terminal value is being successively halved or doubled

12. Big-O Example algorithm • O(n) Printing all elements of an array; • searching an unordered array; • A loop which executes 1 to N times • (where N is the input size, • the number of data values being processed) • O(n log n) Faster sorts; • A loop whose index is being halved/doubled • inside a loop executing from 1 to N

13. Big-O Example algorithm • O(n2) Slower sorts; • A loop which executes from 1 to N times • inside a loop executing from 1 to N times • O(n3) Incrementing all the elements in a NxNxN array; • A 1..N loop inside a 1..N loop inside a 1..N loop • O(nk) k levels of loops inside loops

14. Big-O Example algorithm • O(2n) List all the subsets of a set with n elements; • practical only with small values of n; • n=64 takes 5 years on a supercomputer • O(n!) List all the possible arrangements of a set with n elements

15. Big-O General Rules • a.) Multiplicative constants do not matter. O(3n) = O(n) • b.) Addition is done by taking the max. O(n2) + O(n) = O(max(n2, n)) = O(n2) O(n3) + O(log n) = O(max(n3, log n)) = O(n3) • c.) Multiplication remains multiplication. O(n) * O(log n) = O(n log n) O(n) * O(n2) = O(n3) • d.) Lower is always O(higher) for n³1, but closest is desired f(n2) = O(n3)

16. How can a program with many control structures be categorized by only one? • A certain country taxes pet stores based only on the combined weight of all the pets. • Only two pets thrive in the country: parakeets and elephants • When reporting their taxes, pet shop owners generally only report the weight of the _____.

17. Example Complexities • 4n3 + 20n + 30 = O(n3) • n + 10000 = O(n) • 4n4 + 20n + 30 = O(n4) • 2n + n3 = O(2n) • 200 = O(1)

18. Running Time of Statements • Simple statements are: O(1) • initialization of variables; input/ouput • Loops are: O(g(n)f(n)) • where • g(n) is upper bound on number of loop iterations & • f(n) is upper bound on the body of the loop • If g(n) and f(n) are constant, then this is constant time

19. Running Time of Statements (continued) • Conditional statements are O(max(f(n),g(n))) • where • f(n) is upper bound on then part and • g(n) is upper bound on else part • Blocks of statements with complexities: f1(n), f2(n), ..,fk(n), • have complexity O(f1(n) + f2(n)+ ...+ fk(n))

20. Simple Analysis cin >> n; // 1 if (n > 20) // 1 cout << “Number is > 20” << endl; // 2 else cout << “Number is <= 20” << endl; // 2 T(n) = 2 + max (2,2) = 4 = O(1)

21. Example Analysis cin >> n; // 1 factorial = 1; // 1 for (i = 2; i<=n; i++) // 1 initialization +1 test + //(1 test + 1 inc) per iteration factorial *= i; // 1 cout << factorial; // 1 T(n) = 5 + 3*n = O(n)

22. Another example cin >> n; // 1 if (n > 0) { // 1 factorial = 1; // 1 for (i = 2; i <=n; i++) // 1 initialization +1 test + //(1 test + 1 inc) per iteration factorial *= i; // 1 cout << factorial; // 1 } else cout << “Can’t do factorial of” << n; // 2 T(n) = 2 + max(4+3*n, 2) = O(n)

23. Analysis of simple function calls int factorial(int n) { int fact=1,i; for (i = 2; i <=n; i++) fact *= i; return fact; } void main() { int n; // 1 cin >> n; // 1 cout << factorial(n) << endl; //O(n) } Main is O(n)

24. Analysis of Nested Loops EXAMPLE 1: for (int i = 0; i < n; i++) // n*O(n) = O(n2) for (int j=0; j<n;j++) // n*O(1) = O(n) x++; // O(1) EXAMPLE 2: for (int i = 0; i < n; i++) for (int j=i; j<n;j++) x++; // O(1) T(n) = n+(n-1) + (n-2)+…+ 1 = n(n+1)/2 = O(n2)

25. Summing blocks of statements for (int i = 0; i < n; i++) // O(n2) for (int j=0; j<n;j++) x++; for (int i = 0; i < n; i++) // O(n) x++; T(n) = O(n2) + O(n) = O(n2 + n) = O(n2)

26. Loops with different limits for (int i = 0; i < n; i++) // n*O(m) = O(mn) for (int j=0; j<m; j++) // m*O(1) = O(m) x++; // O(1) O(mn)

27. More complex loops for (i = 1;i < n; i *=2) x++; for (i=n; i > 0; i/=2) x++; Repetitive halving or doubling results in logarithmic complexity. Both loops are O(log n)

28. The Integers and Division 2.4

29. Divides (|), Prime • Number theory - the study of integers and their properties • a | b (a divides b) - $ integer c where ac = b • 3 | 12 since 12 = 3 * 4 • Prime - an integer > 1 whose only factors are the number and 1 • Composite - an integer > 1 that is not prime • Fundamental theorem of arithmetic • Every positive int. can be written as a product of primes • 100 = 2 * 2 * 5 * 5 • If n is composite, it has a prime divisor £

30. Divisibility Tests • A number is divisible by 2 if its unit’s digit is divisible by 2. • A number is divisible by 3 if the sum of the digits is divisible by 3. • A number is divisible by 4 if the # made of the last two digits is div by 4. • A number is divisible by 5 if its unit’s digit is divisible by 5. • A number is divisible by 6 if its is divisible by 2 and 3. • A number is divisible by 8 if the # made of the last 3 digits is div by 8. • A number is divisible by 9 if the sum of the digits is divisible by 9.

31. The Division Algorithm Given integer a and positive integer d. • $ integers q,r with 0 £ r < d, such that a = dq + r • d is divisor, r remainder, q quotient, a dividend quotient remainder divisordividend 101 divided by 11 == quotient 9, remainder 2 101 = 11* 9 + 2 -11 divided by 3 == quotient -4, remainder 1 -11 = 3(-4) + 1

32. Greatest Common Divisor • gcd(a,b) = largest integer d, such that d | a and d | b gcd(24,36) = 12 • Algorithm for finding gcd(a,b) • Express a, b as powers of products of primes • The products of lowest powers of all distinct primes is gcd(intersection) • gcd(252,420) • 252 = 22 ×32×7420 = 22 ×3 ×5 ×7 • gcd(252,420) = 22 ×3 ×7 = 84

33. Relatively Prime • Two integers are relatively prime if their gcd is 1 • 17 and 22 are r.p. • 6 and 22 are not r.p. • The integers a1, a2, …, an are pairwise relatively prime if gcd(ai,aj) = 1, whenever 1 £ i < j £ n • 10, 17 and 21 are p.r.p. • 10, 17 and 22 are not p.r.p. because gcd(10,22) = 2

34. Least Common Multiple • lcm(a,b) = smallest positive integer that is divisible by both a and b • Finding lcm(a,b) • Express a, b as powers of products of primes • The products of highest powers of all distinct primes is lcm (union) • lcm(252,420) • 252 = 22 ×32×7420 = 22 ×3 ×5 ×7 • gcd(252,420) = 22 ×32×5 ×7 = 1260 • NOTE: ab = gcd(a,b)× lcm(a,b)

35. Modular Arithmetic • a mod m • the remainder when a is divided by m 17 mod 5 = 2 2001 mod 101 = 82 19 r 82 • What about negative operands? -17 mod 5 The answer in C++ is –2 (-3 –2/5) Text answer would be 3 (-4 + 3/5) • Mod 0 is not defined

36. Congruence • a º b (mod m) a Is Congruent to b modulo m • a mod m = b mod m OR • m divides a-b • 17 º5 (mod 6) is true 6 divides (17 - 5) 17 mod 6 = 5 5 mod 6 = 5 • 24 º14 (mod 6) is false 6 does not divide (24 - 14) 24 mod 6 = 0 14 mod 6 = 2

37. Hashing • Applying a function to a key value • mapping range of possible key values • into a smaller range of addresses • h(k) = k mod m • h(064212848) = 064212848 mod 111 = 14 • h(037149212) = 037149212 mod 111 = 65

38. Cryptology • The study of secret messages • Caesar cipher • f (p) = (p + 3) mod 26 • “MEET YOU IN THE PARK” becomes • “PHHW BRX LQ WKH SDUN”

39. Security Now Podcast Leo LaPorte, twit.tv, leoville.com Steve Gibson, grc.com

40. Decoder Ring (SN-31)

41. Basic Terminology plaintext - the original message ciphertext - the coded message cipher - algorithm for transforming plaintext to ciphertext key - info used in cipher known only to sender/receiver encipher (encrypt) - converting plaintext to ciphertext decipher (decrypt) - recovering ciphertext from plaintext cryptography - study of encryption principles/methods cryptanalysis (codebreaking) - the study of principles/ methods of deciphering ciphertext without knowing key cryptology - the field of both cryptography and cryptanalysis

42. Cryptology • The study of secret messages • Caesar cipher • f (p) = (p + 3) mod 26 • “MEET YOU IN THE PARK” becomes • “PHHW BRX LQ WKH SDUN”

43. English Letter Frequencies

44. Letter Pair Frequencies • For instance, given a section of English language, E tends to be very common, while X is very rare. Likewise, ST, NG, TH, and QU are common pairs of letters (termed bigrams or digraphs), while NZ and QJ are rare. The mnemonics phrase "ETAOIN SHRDLU" encodes the 12 most frequent letters in typical English language text.

45. EXCLUSIVE OR The EXCLUSIVE OR is true when exactly one of its operands is true. p q p Å q ____________________ F F F F T T T F T T T F

46. Reversible XOR 1100 plaintext Å 1010 encrypt ----- 0110 ciphertext Å 1010 decrypt ----- 1100 plaintext

47. Random Numbers • Generating a sequence of random numbers is often useful • In a game, it ensures that a player does not seethe same behavior each time • In a simulation of a complex system,random numbers can be used tohelp generate random events • Car crash in a simulationof a highway system • Likelihood of a gene in cell mutation • Weather simulation

48. Uniform Random Numbers • Uniform random number sequence • A sequence of random numbers where • Each value in the sequence is drawn from the same range of numbers • In each position of the sequence, any value in the number range is equally likely to occur

49. Random Numbers • Examples • Generate a uniform randomnumber sequence in the range1 to 6 • Use a fair six-sided dice • Each roll represents a new random number • Do two die produce uniform random numbers in the range 1 ... 12? • Generate a uniform random numbersequence in the range 1 to 2 • Use a fair coin • Heads: 1, Tails: 2

50. Random Numbers • We can write an algorithmfor generating what lookslike random numbers • Because it’s an algorithm,we know the rules for generating the next number • The generated numbers are not really random • They are properly called pseudorandom numbers 30 21 9 28 29 ...