Design and Analysis of Algorithms

1. Design and Analysis of Algorithms

2. Textbook Cover What does the cover include? Puzzles, Notions, and Stories of Algorithms.

3. Text Book and Reference Books • Text Book: • Anany Levitin, Introduction to the Design and Analysis of Algorithms, Tsinghua University Press, 2003, 39RMB. • Reference Books: • Thomas H. Cormen etc., Introduction to Algorithms (Second Edition), Higher Education Press & The MIT Press. • Others, search on the website with keyword “Design and Analysis of algorithm”, you will find more...

4. Grading Schemes • Total 100 • Class participation and Assignments: 10% (Individual) • Midterm Test I: 15% (Individual) • Midterm Test II: 15% (Individual) • Labs : 20% (Individual) • Final Exam: 40% (Individual) • Bonus is possible for those positive and active students

5. Caution • Please finish your assignments and projects independently. • If you really meet problems, you can seek help from me or other students, but remember not copying, CHEATING! • Feedback (positive or negative) is encouraged and welcome, and IMPORTANT. (ask questions!) • Thinking, before, in and after class is encouraged and IMPORTANT.

6. Why Study this Course? • Donald E. Knuth stated “Computer Science is the study of algorithms” • Cornerstone of computer science. Programs will not exist without algorithms. • Closely related to our lives • Help to guide how others analyze and solve problems • Help to develop the ability of analyzing and solving problems via computers. • Very interesting if you can concentrate on this course

7. Course Prerequisite • Data Structure • Discrete Mathematics • C or other programming languages • Advanced Mathematics

8. Algorithm: A brief History • Muhammad ibn Musa al-Khwarizmi, one of the most influential mathematicians in 9thcenturyin Baghdad, wrote a textbook in Arabic about adding, multiplying, dividing numbers, and extracting square roots and computing π. • www.lib.virginia.edu/science/parshall/khwariz.html • Many centuries later, decimal system was adopted in Europe, and the procedures in Al Khwarizmi’s book were named after him as “Algorithms”. (image of Al Khwarizmi from http://jeff560.tripod.com/)

9. What I hear, I forget. What I see, I remember. What I do, I understand. What does this mean? Let’s start our lecture formally 

10. Notion: Algorithm • You all have learned the course DATA STRUCTURE or DISCRETE MATHEMATICS, and have your own sense on algorithm. • to your opinion, what is algorithm?

11. problem algorithm “computer” output input Notion: Algorithms • An algorithm is a sequence of unambiguous instructions for solving a computational problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

12. Notion: Algorithm • More precisely, an algorithm is a method or process to solve a problem satisfying the following properties: • Finiteness • terminates after a finite number of steps • Definiteness • Each step must be rigorously and unambiguously specified. -e.g., ”stir until lumpy” • Input • Valid inputs must be clearly specified. • Output • can be proved to produce the correct output given a valid input. • Effectiveness • Steps must be sufficiently simple and basic.

13. Examples of Algorithms – Computing the Greatest Common Divisor of Two Integers • gcd(m, n): the largest integer that divides both m and n. • First try -- Euclid’s algorithm:gcd(m, n) = gcd(n, m mod n) • Step1: If n = 0, return the value of m as the answer and stop; otherwise, proceed to Step 2. • Step2: Divide m by n and assign the value of the remainder to r. • Step 3: Assign the value of n to m and the value of r to n. Go to Step 1.

14. Example: So, gcd(1785,546)=21

15. Methods of Specifying an Algorithm • Natural language • Ambiguous “Mike ate thesandwich on a bed.” • Pseudocode • A mixture of a natural language and programming language-like structures • Precise and succinct. • Pseudocode in this course • omits declarations of variables • use indentation to show the scope of such statements as for, if, and while. • use  for assignment

16. Pseudocode of Euclid’s Algorithm AlgorithmEuclid(m, n) //Computes gcd(m, n) by Euclid’s algorithm //Input: Two nonnegative, not-both-zero integers m and n //Output: Greatest common divisor of m and n while n ≠ 0 do r  m mod n m  n n  r return m • Questions: • Finiteness: how do we know that Euclid’s algorithm actually comes to a stop? • Definiteness: nonambiguity • Effectiveness: effectively computable.

17. Second Try for gcd(m, n) • Consecutive Integer Algorithm • Step1: Assign the value of min{m, n} to t. • Step2: Divide m by t. If the remainder of this division is 0, go to Step3;otherwise, go to Step 4. • Step3: Divide n by t. If the remainder of this division is 0, return the value of t as the answer and stop; otherwise, proceed to Step4. • Step4: Decrease the value of t by 1. Go to Step2. • Questions • Finiteness • Definiteness • Effectiveness • Which algorithm is faster, the Euclid’s or this one?

18. Consecutive Integer Algorithm Find gcd(9,6)? Step1: t=min{9,6} (t=6) Step2: s m%tn%t 6 9%6=3 6%6=0 5 9%5=4 6%5=1 4 9%4=5 6%4=2 3 9%3=0 6%3=0

19. Third try for gcd(m, n) • Middle-school procedure • Step1: Find the prime factors of m. • Step2: Find the prime factors of n. • Step3: Identify all the common factors in the two prime expansions found in Step1 and Step2. (If p is a common factor occurring Pm and Pn times in m and n, respectively, it should be repeated in min{Pm, Pn} times.) • Step4: Compute the product of all the common factors and return it as the gcd of the numbers given. • Question • Is this a legitimate algorithm?

20. Sieve of Eratosthenes Input: Integer n ≥ 2 Output: List of primes less than or equal to n for p ← 2 to n do A[p] ← p for p ← 2 to sqrt(n) do if A[p]  0 //p hasn’t been previously eliminated from the listj ← p*p while j ≤ ndo A[j] ← 0 //mark element as eliminated j ← j+ p Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

21. What can we learn from the previous 3 examples? • Each step of an algorithm must be unambiguous. • The same algorithm can be represented in several different ways. (different pseudocodes) • There might exists more than one algorithm for a certain problem. • Algorithms for the same problem can be based on very different ideas and can solve the problem with dramatically different speeds.

22. Fundamentals of Algorithmic Problem Solving • Understanding the problem • Asking questions, do a few examples by hand, think about special cases, etc. • Deciding on • Exact vs. approximate problem solving • Appropriate data structure • Design an algorithm • Proving correctness • Analyzing an algorithm • Time efficiency : how fast the algorithm runs • Space efficiency: how much extra memory the algorithm needs. • Coding an algorithm

23. Algorithm Design and Analysis Process

24. Two main issues related to algorithms • How to design algorithms • How to analyze algorithm efficiency

25. Brute force Divide and conquer Decrease and conquer Transform and conquer Space and time tradeoffs Algorithm Design Techniques/Strategies • Greedy approach • Dynamic programming • Backtracking • Branch and bound

26. Analysis of Algorithms • How good is the algorithm? • time efficiency • space efficiency • Does there exist a better algorithm? • lower bounds • optimality

27. Example: Fibonacci Number • Fibonacci’s original question: • Suppose that you are given a newly-born pair of rabbits, one male, one female. • Rabbits are able to mate at the age of one month so that at the end of its second month a female can produce another pair of rabbits. • Suppose that our rabbits never die. • Suppose that the female always produces one new pair (one male, one female) every month. • Question: How many pairs will there be in one year? • 1. In the beginning: (1 pair) • 2. End of month 1: (1 pair) Rabbits are ready to mate. • 3. End of month 2: (2 pairs) A new pair of rabbits are born. • 4. End of month 3: (3 pairs) A new pair and two old pairs. • 5. End of month 4: (5 pairs) ... • 6. End of month 5: (8 pairs) ... • 7. after 12 months, there will be 233 rabits (image of Leonardo Fibonacci from http://www.math.ethz.ch/fibonacci)

28. Recurrence Relation of Fibonacci Number fib(n): {0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, …}

29. Algorithm: Fibonacci Problem: What is fib(200)? What about fib(n), where n is any positive integer? Algorithm 1 fib(n) if n = 0 then return (0) if n = 1then return (1) return (fib(n − 1) + fib(n − 2)) Questions that we should ask ourselves. 1. Is the algorithm correct? 2. What is the running time of our algorithm? 3. Can we do better?

30. Answer of Questions • Is the algorithm correct? • Yes, we simply follow the definition of Fibonacci numbers • How fast is the algorithm? • If we let the run time of fib(n) be T(n), then we can formulate T(n) = T(n − 1) + T(n − 2) + 3  1.6n • T(200)  2139 • The world fastest computer BlueGene/L, which can run 248 instructions per second, will take 291 seconds to compute. (291 seconds = 7.85 × 1019 years ) • Can Moose’s law, which predicts that CPU get 1.6 times faster each year, solve our problem? • No, because the time needed to compute fib(n) also have the same “growth” rate • if we can compute fib(100) in exactly a year, • then in the next year, we will still spend a year to compute fib(101) • if we want to compute fib(200) within a year, we need to wait for 100 years.

31. Improvement • Can we do better? • Yes, because many computations in the previous algorithm are repeated. Algorithm 2: fib(n) comment: Initially we create an array A[0: n] A[0] ← 0,A[1] ← 1 for i ← 2 to n do A[i] ← A[i − 1] + A[i − 2] return (A[n])

32. Important problem types • sorting • searching • string processing • graph problems • combinatorial problems • geometric problems • numerical problems

33. list array linked list string stack queue priority queue Fundamental data structures • graph • tree • set and dictionary