CS301 - Algorithms

1. CS301 - Algorithms Fall 2006-2007 Hüsnü Yenigün

2. Contents • About the course • Introduction • What is an algorithm? • Computational problems • An instance of a problem • Correctness of algorithms • Loop invariant method for showing correctness of algorithms • What does a better algorithm mean? • Selecting the best algorithm • Analysis of “Insertion Sort” algorithm • Which running time – best, average, worst – should we use? • Asymptotic analysis • Divide and Conquer • Analysis of divide and conquer algorithms • Growth of functions & Asymptotic notation • O-notation (upper bounds) • o-notation (upper bounds that are not tight) • Ω-notation (lower bounds) • ω-notation (lower bounds that are not tight) • Θ-notation (tight bounds) • Some properties of asymptotic notations • Some complexity classes CS301 – Algorithms [ Fall 2006-2007 ]

3. About the course • Instructor • Name: Hüsnü Yenigün • Office: FENS 2094 • Office Hours: • Walk-in: Thursdays 08:40-09:30 • By appointment: any available time  (please read the footnote) http://people.sabanciuniv.edu/yenigun/calendar.php • TAs • Name: Mahir Can Doğanay, Ekrem Serin • Office: FENS xxxx • Office Hours: to be decided CS301 – Algorithms [ Fall 2006-2007 ]

4. About the course Weights of these grading items will be announced at the end of the semester. • Evaluation: • Exams: • 1 Midterm (November 23, 2006 @ 09:40 – pending approval) • 1 Final (to be announced by Student Resources) • 1 Make-up (after the final exam) • You have to take exactly 2 of these 3 exams (no questions asked about the missed exam) - Make-up, if taken, counts as the missed exam - If you take only one or none of these exams, the missing exam/exams is/are considered to be 0 • Homework~ 6-7 CS301 – Algorithms [ Fall 2006-2007 ]

5. About the course • Recitations: • Problem solving • Not regular • Will be announced in advance • All communication through WebCT • No e-mail to my personal account (you can send me an e-mail to let me know that you’ve posted something on WebCT) CS301 – Algorithms [ Fall 2006-2007 ]

6. About the course • Course material • Textbook: Introduction to Algorithms by Cormen et al. • Lecture notes: ppt slides (will be made available on WebCT) CS301 – Algorithms [ Fall 2006-2007 ]

7. Why take this course? • Very basic – especially for CS and MSIE – and intellectually enlightening course • Get to know some common computational problems and their existing solutions • Get familiar with algorithm design techniques (that will help you come up with algorithms on your own) • Get familiar with algorithm analysis techniques (that will help you analyze algorithms to pick the most suitable algorithm) • Computers are not infinitely fast and memory is limited • Get familiar with typical problems and learn the bounds of algorithms (undecidability and NP-completeness) CS301 – Algorithms [ Fall 2006-2007 ]

8. Tentative Outline • Introduction • Asymptotic Notation • Divide and Conquer Paradigm • Recurrences • Solving Recurrences • Quicksort • Sorting in linear time • Medians and order statistics • Binary search trees • Red-Black trees • Augmenting data structures • Dynamic programming • Greedy algorithms • Amortized Analysis • B-Trees • Graph algorithms • Sorting Networks • Computational Geometry • Undecidability • NP-Completeness CS301 – Algorithms [ Fall 2006-2007 ]

9. INTRODUCTION CS301 – Algorithms [ Fall 2006-2007 ]

10. What is an algorithm? Sequence of trivial steps • An algorithm is a well-defined computational procedure that takes a value (or a set of values) as input, and produces a value (or a set of values) as output, as a solution to a computational problem. Algorithm output input CS301 – Algorithms [ Fall 2006-2007 ]

11. The statement of the problem defines what is the relationship between the input and the output. • The algorithm defines a specific computational procedure that explains how this relationship will be realized. CS301 – Algorithms [ Fall 2006-2007 ]

12. An example computational problem… • Given a function find a surjection such that • This is nothing but a formal definition of the sorting problem (the problem as described above asks for an algorithm that sorts the input numbers in nondecreasing order) CS301 – Algorithms [ Fall 2006-2007 ]

13. An example computational problem… • The problem definition is not always given as formal as in the previous example: CS301 – Algorithms [ Fall 2006-2007 ]

14. Sorting example • Given a sequence of numbers as input such as [ 15, 42, 17, 34, 3, 17 ] • The output should be [ 3, 15, 17, 17, 34, 42 ] • Note that, the output for this input is in accordance with the problem definition, i.e. it conforms with the “what should be done” definition given in the problem statement . • “How it should be done” depends on the algorithm. CS301 – Algorithms [ Fall 2006-2007 ]

15. An instance of a problem • Aninstance of a problem consists of all the inputs that satisfy the constraints that are imposed by the problem definition. • “Sort [15, 42, 17, 34, 3, 17] in nondecreasing order” is an instance of the sorting problem. • The input is a sequence of numbers (not a sequence of letters, or a set of numbers). CS301 – Algorithms [ Fall 2006-2007 ]

16. Not the sorting problem again !!! • Sorting is a fundamental operation in many disciplines and it is used as a part of other algorithms. • A lot of research has been made on the sorting problem. • A lot of algorithms have been developed. • It is a very simple and interesting problem to explain basic ideas of algorithm design and analysis techniques. CS301 – Algorithms [ Fall 2006-2007 ]

17. Correctness of algorithms • An algorithm is correct if for every instance of the problem, it halts (terminates) producing the correct answer. • Otherwise (i.e. if there are some instances for which the algorithm does not halt, or it produces an incorrect answer), it is called an incorrect algorithm. • Surprisingly, incorrect algorithms are occasionally used in practice (e.g. primes problem)… CS301 – Algorithms [ Fall 2006-2007 ]

18. Insertion sort • Basic idea: Given a nondecreasing sequence [a1, a2, …, an] and a number k the sequence [a1, a2, … aj, k, aj+1,…, an] is a nondecreasing sequence if aj ≤ k≤aj+1 • For example: [a1, a2, a3, a4, a5] k [10, 12, 22, 34, 35] 19 the result is: [10,12,19,22,34,35] CS301 – Algorithms [ Fall 2006-2007 ]

19. [1,3] [1,3,7] Insertion sort • How can we use this idea to sort a sequence of numbers? • Suppose we are given: [ 3, 1, 7, 2 ] • Start with a single element sequence (it is already a sorted sequence) • Insert each element one-by-one into already sorted sequence. • It is like sorting a hand of a card (e.g. bridge) game… [3] [1,2,3,7] CS301 – Algorithms [ Fall 2006-2007 ]

20. Pseude code for Insertion sort Considers each element one-by-one. Note that, the first element is assumed to form the initial sorted sequence. Searches for the correct place to insert the next element. Insertion-Sort(A) { for (j=2; j≤n; j=j+1) { num = A[j]; i = j-1; // find the correct place for num while (i>0 and A[i]>num) { A[i+1] = A[i]; i=i-1; } A[i+1] = num; } } If it sees that num is smaller than A[i], it shifts A[i] one position to the right When the correct place is found, it will be already empty CS301 – Algorithms [ Fall 2006-2007 ]

21. BREAK CS301 – Algorithms [ Fall 2006-2007 ]

22. Showing the correctness of Insertion Sort • Note that, “Insertion-Sort” is an iterative algorithm. • Loop invariants is a widely used method to show the correctness of iterative algorithms. • A loop invariant is a boolean statement that is correct for all the iterations of the loop. • Loop invariant method is performed in 3 steps, and related to the mathematical induction proofs. CS301 – Algorithms [ Fall 2006-2007 ]

23. 3 steps of loop invariants method • Initialization: Show that the loop invariant holds before the first iteration of the loop. • Maintenance: Show that if the loop invariant holds before an iteration of the loop, then it also holds after the next iteration of the loop. • Termination: When the loop terminates, the invariant gives a useful property that helps to show that the algorithms is correct. Similar to the “induction base” in inductive proofs Similar to “induction step” in inductive proofs CS301 – Algorithms [ Fall 2006-2007 ]

24. A loop invariant for Insertion sort • A loop invariant of the “for loop” of the insertion sort: “The sub-array A[1..j-1] holds nondecreasing sequence of numbers” CS301 – Algorithms [ Fall 2006-2007 ]

25. Step 1 • Initially, when the loop starts its first iteration we have j=2 • Therefore, initially A[1..j-1] = A[1..2-1] = A[1..1] • Since A[1..1] is a single element subarray, it is a sorted sequence numbers. • Hence, the loop invariant initially holds. CS301 – Algorithms [ Fall 2006-2007 ]

26. Step 2 • Assume the loop invariant holds just before an iteration. • Within the iteration, the while loop will shift all the numbers that are strictly greater than A[j] one slot to the right. • A[j] will be inserted after all the elements that are smaller or equal to A[j] • Hence after the iteration finished, A[1..j-1] will be a sorted sequence. CS301 – Algorithms [ Fall 2006-2007 ]

27. Step 3 • When the algorithm terminates, it means j > n • Since we increment j by 1 in each iteration, we know that j=n+1. • The loop invariant for this value of j states that: A[1..j-1]=A[1..n+1-1]=A[1..n] is a sorted sequence of numbers. QED CS301 – Algorithms [ Fall 2006-2007 ]

28. Is Insertion sort the solution for the sorting problem? • Insertion sort is only a solution for the sorting problem. • “But we’ve just proved that it works correctly for all the input sequences. Why do we need other algorithms to solve the sorting problem?” • There may be other algorithms better than Insertion sort… CS301 – Algorithms [ Fall 2006-2007 ]

29. What does a “better algorithm” mean? • A better algorithm uses less resources than the other algorithms. • Then, just show us the best algorithm known. We will only be using the best algorithm. • Not that simple. Using less resource depends on • The number of input elements • The characteristics of the input • So, the definition of “best” changes • Time (*) • Space • Money • Area • Bandwidth • etc. CS301 – Algorithms [ Fall 2006-2007 ]

30. Selecting the best algorithm • Selection the best algorithm, first of all, requires to have multiple algorithms for the solution of the same problem. • The resource usage on which our selection will be made should be known. • And, we must analyze the available algorithms to understand how much of the type of resource we are interested these algorithms use. • We must have a specific model of implementation for the analysis. We will mainly use the RAM (random access machine) model, where the algorithms are implemented as computer programs. In RAM model, statements are executed one by one. CS301 – Algorithms [ Fall 2006-2007 ]

31. Analysis of Insertion sort • Time taken by Insertion sort depends on • The number of elements to be sorted 10 elements vs. 1000 elements • The nature of the input already sorted vs. reverse sorted • In general, the time taken by an algorithm grows with the size of the input. • Therefore, we describe the running time of an algorithm as a function of the input size. CS301 – Algorithms [ Fall 2006-2007 ]

32. Definition of the input size • It depends on the problem. • For sorting problem, it is natural to pick the number of elements as the size of the input. • For some problems, a single measure is not sufficient to describe the size of the input. • For example, for a graph algorithm, the size of the graph is better described with the number of nodes and the number of edges given together. CS301 – Algorithms [ Fall 2006-2007 ]

33. Definition of the running time • We can use a 1990 PC AT computer or a contemporary supercomputer to execute an implementation of the algorithm. • A good programmer can implement the algorithm directly using assembly code, or a beginner programmer can implement it using a high level language and compile it using the worst compiler (which has no optimization). • So, the running time of a given algorithm seems to depend on certain conditions. CS301 – Algorithms [ Fall 2006-2007 ]

34. Definition of the running time • Our notion of “running time” should be as independent as possible from such consideration. • We will consider the “number of steps” on a particular input as the running time of an algorithm. • For the time being, let us assume that each step takes a constant amount of time. CS301 – Algorithms [ Fall 2006-2007 ]

35. Running time of insertion sort cost times executed Insertion-Sort(A) { for (j=2; j≤n; j=j+1) { num = A[j]; i = j-1; // find the correct place for num while (i>0 and A[i]>num) { A[i+1] = A[i]; i=i-1; } A[i+1] = num; } } c1 n c2 n-1 c3 n-1 c4 c5 c6 c7 n-1 kj: the number of times the “while” loop condition is checked for that specific j value CS301 – Algorithms [ Fall 2006-2007 ]

36. Running time of insertion sort • The total running time can be calculated as: • With a little bit of calculation: CS301 – Algorithms [ Fall 2006-2007 ]

37. Running time of insertion sort (best case) • Recall that kj is the number of times that the “while loop” condition is checked to find the correct place of a number • Under the best scenario, it will never iterate for all j, hence kj =1 for all j • This corresponds to the case where the input is already sorted • In this case CS301 – Algorithms [ Fall 2006-2007 ]

38. Running time of insertion sort (worst case) • Under the worst scenario, the while loop will iterate the maximum amount of time possible • Therefore, kj = j for all j CS301 – Algorithms [ Fall 2006-2007 ]

39. Running time of insertion sort (average case) • On the average, the while loop will iterate half of the maximum amount of time possible • Therefore, kj = j/2 for all j CS301 – Algorithms [ Fall 2006-2007 ]

40. Running time of insertion sort • Best case: Linear function of n • Average case: Quadratic function of n • Worst case: Quadratic function of n CS301 – Algorithms [ Fall 2006-2007 ]

41. BREAK CS301 – Algorithms [ Fall 2006-2007 ]

42. Which running time we should use? • In order to compare the running time of algorithms, usually the “worst case running time” is used, because • It gives an upper bound (it cannot go worse) • Murphy’s law (most of the time, the worst case appears) • Average case is usually the same as the worst case. CS301 – Algorithms [ Fall 2006-2007 ]

43. Asymptotic Analysis • Note that, in the running time analysis of the insertion sort algorithm, we ignored the actual cost of steps by abstracting them with constants : ci • We will go one step further, and show that these constants are not actually so important. CS301 – Algorithms [ Fall 2006-2007 ]

44. Asymptotic Analysis • Suppose we have two algorithms for sorting A1 and A2 • Let the exact running time of them be • Assume A1 is executed on a fast machine (109 instructions per second) • Assume A2 is executed on a slow machine (106 instructions per second) • Assume we will be sorting 105 numbers CS301 – Algorithms [ Fall 2006-2007 ]

45. Asymptotic Analysis • A1 on the fast computer will need • A2 on the slow computer will need A2 will run four times faster CS301 – Algorithms [ Fall 2006-2007 ]

46. Asymptotic Analysis • In real life, we will be interested in the performance of the algorithms on large inputs. • Therefore, even if the coefficients of the exact running time are small, it is the growth of the function (highest order term) that determines the performance of the algorithms as the input size gets bigger. CS301 – Algorithms [ Fall 2006-2007 ]

47. Asymptotic Analysis • Look at growth of T(n) as n→ ∞ • Θ-notation: • Ignore lower order terms • Ignore leading constants • For example: Leading constant Lower order terms CS301 – Algorithms [ Fall 2006-2007 ]

48. Asymptotic Analysis time input size CS301 – Algorithms [ Fall 2006-2007 ]

49. Algorithm Design Techniques • In general, there is no recipe for coming up with an algorithm for a given problem. • However, there are some algorithms design techniques that can be used to classify the algorithms. • Insertion sort uses so called “incremental approach” Having sorted A[1..j-1], insert a new element A[j], forming a new, larger sorted sequence A[1..j] CS301 – Algorithms [ Fall 2006-2007 ]

50. Divide and Conquer • Another such design approach is Divide and Conquer • We will examine another algorithm for the sorting problem that follows divide and conquer approach • Divide and conquer algorithms are recursive in their nature. • It is relatively easy to analyze their running time CS301 – Algorithms [ Fall 2006-2007 ]