CS305/503, Spring 2009 Advanced Sorting

1. CS305/503, Spring 2009Advanced Sorting Michael Barnathan

2. Warning • We were sort of warming up until now. Now we get to the hard part. • The topics here are no longer simple. • We will use recursion in a complex manner. • We will mathematically formalize a method of analyzing recursive algorithms. • We will be writing some tricky code. • With some exceptions, they will only continue to grow more complex, through this course and through your curriculum. • If you are not secure with the Java language by now, please make an effort to learn it ASAP. • You will not be able to complete Assignment 3 otherwise. • I’m going to present more code in class than I have been, but the emphasis still needs to remain on the data structures. • Try some coding problems from the book. • See Alex or another tutor. • See me during office hours (I’ve only had two students visit).

3. Here’s what we’ll be learning: • Theory: • Shellsort. • Mergesort. • Quicksort. • Big Omega and Theta. • Analyzing recursive algorithms: the master method. • Comparison of basic and advanced sorting algorithms. • Java: • How to create generic methods.

4. Basic Sorting Algorithms • Bubble Sort • Loop through the array twice, “bubble” elements up the array one-by-one by swapping until each reaches where it belongs. • Advantages: • Simplicity. • Stable. • Requires O(1) extra space. • It has a cool name. • Disadvantages: • Asymptotic Performance: O(n^2). • Real-world performance: ~ Twice as slow as insertion sort.

5. Basic Sorting Algorithms • Insertion Sort • Build a sorted “buffer” within the array. Gradually expand the buffer by swapping elements into their appropriate places within it. • Advantages: • Stable. • Requires O(1) extra space. • Usually the fastest basic sorting algorithm. • Disadvantages: • Asymptotic Performance: O(n^2). • More complex than bubble and selection sort.

6. Basic Sorting Algorithms • Selection Sort • Keep a buffer. Search for the minimum element outside of the buffer and swap it with the element at the new end of the buffer (cur. end + 1). • Advantages: • Requires O(1) extra space. • Simple to implement. • Disadvantages: • Asymptotic Performance: O(n^2). • Not as good as insertion sort in practice.

7. Performance • The O(n^2) complexity is a recurring theme. • O(n^2) isn’t that good. • We’d like to do better. • Can we? • Obviously, or we wouldn’t be discussing other algorithms. • But how? • Any ideas? • It’s not really intuitive anymore. • We’re trading performance against simplicity.

8. Lower Bounds and Big Omega • It turns out that any sort that compares adjacent items has a lower bound of Ω(n2). • Huh? Your “O” looks a bit funny. • Reminder: Big O is the most used, but there are other symbols too… • O(n) represents an asymptotic upper bound of n. • Ω(n) represents an asymptotic lower bound of n. • Θ(n) represents both – a “tight bound”. • So when we say adjacent comparison sorts are Ω(n2), we mean that they can be O(n3), O(n4), O(2n), … - but not O(n log n) or better. • This formalizes the notion of “never better than this”. • Likewise, that means an algorithm believed to be O(n2) could technically be Θ(n). • However, a tighter O(n) bound would have to be proven for this. • Big O formalizes the notion of “never worse than this”. • We use this one the most because computer scientists are pessimists. • If you know for sure that your algorithm will perform in time proportional to n2 and NOT proportional to n or n3, it’s best to say Θ(n2). • If you’re not sure whether it runs in time proportional to n or n2, you can just leave it at O(n2) – someone might later improve the tightness of your upper bound.

9. Wait • Don’t confuse this with best/average case performance. • You’re still analyzing the worst case. • Using O(n) or Ω(n) reflects the idea of doing (no worse or no better) in the worst case. • Loose bounds are more about how well we know this algorithm’s behavior rather than how it necessarily behaves. • With more knowledge, we could tighten the bound.

10. Comparison

11. Shell Sort • Insertion sort is a good algorithm, but slow. • Why is it slow? • Elements are only swapped one position at a time. • We can jump a bit further and perhaps gain some performance. • But we just said we can’t do better if we only compare adjacent elements. • Ok, so don’t compare adjacent ones then. • Shell sort is a generalization of insertion sort. • It was developed by Donald Shell in 1959. • It introduces the notion of a “gap”. • Insertion sort is Shellsort with a gap of 1.

12. Insertion Sort: Reminder • Here’s the code for insertion sort again: public static <Type extends Comparable<Type> > void insertionSort(List<Type> l) { for (int buffer = 1; buffer < l.size(); buffer++) { Type next = l.get(buffer); int shift; for (shift = buffer-1; shift >= 0 && l.get(shift).compareTo(next) > 0; shift--) swap(l, shift, shift+1); l.set(shift+1, next); } }

13. Why we can improve. • If insertion sort just needs to move every element one space over, it becomes O(n). • It just shifts every element over one space on the first swap and it’s done. • Each successive iteration would just compare the element after the buffer to the first one inside. It wouldn’t need to move, so it would exit. • In general, insertion sort tends to perform well when an array is almost sorted.

14. The Idea • Instead of sorting one element at a time, sort over a wide range first, then narrow it down. Gap = 5 Gap = 2 Gap = 1

15. Maybe it’s simpler with numbers. • [ 1 5 3 7 9 12 8 4 17 ] • Turn into a matrix with gap columns: gap = 3 1 5 3 7 9 12 8 4 17 • Sort each column: 1 4 3 7 5 12 8 9 17 • Turn back into a vector: [1 4 3 7 5 12 8 9 17 ] • Repeat with smaller gaps until we reach 1. gap = 2 1 4 3 7 5 9 8 12 17 • [1 4 3 7 5 9 8 12 17 ] • Gap = 1 corresponds to an insertion sort, which finishes.

16. Shell Games • Why does it work? • Each step prepares the algorithm for the next. • The algorithm doesn’t redo work. • If something is 5-sorted and we sort it with a gap of 3, it will then be both 5 and 3 sorted. • Most effective if we choose a gap sequence in which each pair is relatively prime. • That is, they share no factors greater than 1. • This is so they don’t overlap; e.g., 2 and 6 would both sort 18. • The final insertion sort (gap = 1) should be very close to linear. • But you must always end with gap = 1.

17. How to choose the gap sequence. • It’s a bit of an art. There’s no perfect answer. • It affects the performance of the algorithm. • Usually, you use a mathematical function: • E.g. gap = 3*gap + 1 • The initial value is the largest term in this function that is less than the size of the array. • You then reduce the gap as you sort by inverting the function (gap = (gap – 1) / 3).

18. The Algorithm. //Shell sorts the given collection object with the given gap sequence. public static <Type extends Comparable<Type> > void shellSort(List<Type> l, GapFunctorgapsequence) { int gap = 1; //Starting gap. //Precondition: invGap is an inverse and makes the gap smaller. Checking these is a good habit to learn. if (gapsequence.invGap(gapsequence.nextGap(gap)) != gap || gapsequence.nextGap(gap) <= gap) throw new IllegalArgumentException("invGap must be an inverse and must make the gap smaller"); while (gap < l.size()) //Compute the starting gap - as large as possible. gap = gapsequence.nextGap(gap); //We're now one term past the size of l, so we invert before doing anything else. while (gap > 0) { //Since invGap is an inverse, the last gap will be 1. gap = gapsequence.invGap(gap); //Make the gap smaller according to our function. //This is going through the array backwards, but is otherwise identical to our illustration. for (intendpos = gap; endpos < l.size(); endpos++) { //This is our "sorted buffer". Type temp = l.get(endpos); intstartpos; for (startpos = endpos; startpos >= gap && l.get(startpos - gap).compareTo(temp) > 0; startpos -= gap) l.set(startpos, l.get(startpos - gap)); //Shift. l.set(startpos, temp); //New item in the "buffer". } } }

19. Gap Functions Under This Code • Let’s define GapFunctor as an interface: public interface GapFunctor { //Precondition: the gap size must decrease. public intnextGap(intcurgap); //Precondition: this must invert nextGap. public intinvGap(intnextgap); } • And implement our 3n + 1 function using that interface: //It’s usually attributed to Donald Knuth, so we’ll call it KnuthGap. public class KnuthGap implements GapFunctor { public intnextGap(intcurgap) { return curgap * 3 + 1; } public intinvGap(intnextgap) { return (nextgap - 1) / 3; } }

20. Performance • Shellsort is very difficult to analyze. • If you can manage to do a general theoretical analysis of it, you have material for a publication. • Here are some specific results, however:

21. Optimal polynomial-time sort. • Shellsort requires constant space. • It was proven that the Shellsort cannot reach O(n log n). • But it can come pretty close. • It is not a stable sort. • For small arrays, this may be the optimal sort. • On larger ones, Quicksort and Mergesort outperform it.

22. Mergesort • Fast as Shellsort is, it’s still polynomial. • We can do better than that. • Now we come to the “really good” algorithms. • Mergesort. • Developed in 1945 by John von Neumann. • “Divide and conquer” algorithm. • Yes, it uses recursion. • Java uses it to sort Collection classes. • Like Vector and LinkedList.

23. Mergesort: The Idea. • Similar to the idea we came up with for bubble sort, but with a twist. • In bubble sort, we defined an array of size n as an array of size n - 1 plus an element. • Now we define an array as 2 arrays of size n/2. • The idea behind Mergesort is to split the array in half, sort the smaller arrays, and merge them together in a way that keeps them sorted. • Being recursive, the halves are going to get split into quarters, and those into eighths… • Ok, so what’s the base condition? When do we stop?

24. Mergesort: Recursive Structure. • A 1-element array is already in sorted order. • So we use this as our base case. • The recursive step is pretty simple: • Split the list at the middle. • Merge sort the left half. • Merge sort the right half. • The left and right halves are now sorted. • But we need to merge them together to keep them sorted. • Merging two sorted lists isn’t too hard. • Let’s write it on the board.

25. Illustration • Wikipedia has a nice example:

26. Merge: private static <Type extends Comparable<Type> > Vector<Type> merge(List<Type> l1, List<Type> l2) { intmergedsize = l1.size() + l2.size(); Vector<Type> ret = new Vector<Type>(mergedsize); int l1idx = 0; int l2idx = 0; while (l1idx + l2idx < mergedsize) { //Insert the smaller element from either array. //Since both are sorted, advancing one-by-one will keep them that way. if (l2idx >= l2.size() || (l1idx < l1.size() && l1.get(l1idx).compareTo(l2.get(l2idx)) < 0)) ret.add(l1.get(l1idx++)); else ret.add(l2.get(l2idx++)); } return ret; }

27. Mergesort Algorithm. • Merging on two parts of one array is similar. • So you can do it in-place too. • Then you’d need to pass “leftstart”, “rightstart”, and “rightend” indices and work on the array in-place. • You may wish to implement this variation on your own. I’ll leave it up to you. • We split just like we did in binary search. • Left: Start to middle. • Right: Middle + 1 to end.

28. Mergesort Algorithm: public static <Type extends Comparable<Type> > List<Type> mergeSort(List<Type> l) { //Base case: size <= 1. if (l.size() <= 1) return l; //Already sorted. int mid = l.size() / 2; //Reduction: split down the middle. List<Type> left = mergeSort(l.subList(0, mid)); //Left half. List<Type> right = mergeSort(l.subList(mid, l.size())); //Right half. return merge(left, right); }

29. Mergesort Performance • This is a great algorithm. • It’s stable if you use the out-of-place merge. • The in-place merge is not stable. • It works on sequential data structures. • Why is this? Didn’t we say that partitioning algorithms don’t usually work well on sequential structures? • It’s very fast. • How fast? We’ll see in a moment.

30. The Master Method • We need a general way to analyze recursive algorithms. • Recursion can be modeled with recurrence relations. • These are functions defined in terms of themselves. Like recursive functions, they have base cases and reductions. • We’ve seen one already: • F(n) = F(n-1) + F(n-2).

31. Recursion -> Recurrence • You can define a recurrence on the number of elements being split into. • There are two terms: the recursive term and the constant factor. • The recursive term represents the call with a reduced number of elements. • The constant term represents the work in each call. • For example, printTo(n) called printTo(n-1). • So we would model it with T(n) = T(n-1) + O(1). • The constant term is O(1) because printing n is O(1). • This was all the work that we did inside of the function. • If we split into two lists of size n-1 instead of one, we’d have T(n) = 2T(n-1) + O(1). • If we did something linear inside of printTo, we’d have T(n-1) + O(n). • If we split into two halves, as in mergesort, we’d have 2T(n/2)… • And so on.

32. Mergesort Recurrence • We split into two subproblems of size n/2. • “Of size n/2” -> T(n/2). • “Two subproblems” -> 2*T(…). • “Two problems of size n/2” -> 2T(n/2). • What about the other term? • Mergesort itself is O(1), but it calls merge, which is O(n) – it looks through both arrays in sequence. • So our recurrence is T(n) = T(n/2) + O(n). • We can now use an important mathematical tool to figure out the performance directly from this…

33. The Master Theorem • If given a recurrence of the form:T(n) = a*T(n/b) + f(n)(where f(n) represents the work done in each call). • Then the running time of the algorithm modeled by this recursion will be Θ(nlogba)… • *With an exception: • If f(n) is of the same polynomial order as logba, then the running time will be Θ(nlogba * log n).

34. Examples: • T(n) = T(n/2) + O(1): (Binary search) • A = 1, B = 2, log2 1 = 0 O(n0 * log n). • T(n) = 3T(n/5) + O(n): • A = 3, B = 5, log5 3 = 0.68 O(n0.68). • T(n) = 8T(n/2) + O(1): • A = 8, B = 2, log2 8 = 3 O(n3). • T(n) = 8T(n/2) + O(n3): • A = 8, B = 2, log2 8 = 3 O(n3 log n). • T(n) = 2T(n/2) + O(n): (Mergesort). • A = 2, B = 2, log2 2 = 1 ?

35. Mergesort Performance Θ(n log n)! • This blows the O(n2) sorts out of the water. • So mergesort: • Is stable with O(n) extra space. • Works on sequential structures. • Has great worst (and average)-case performance. • Works really well in practice. • Is a great algorithm to use almost anywhere, as long as the O(n) space isn’t a deterrent. • Despite this, it is not usually considered the fastest general purpose sort. • There is one that’s faster…

36. Can we do better than O(n log n)? • It has been proven (using decision trees) that O(n log n) is the optimal complexity for any sort that compares elements. • You might want to seek out the proof; it’s pretty intuitive and not too difficult to comprehend. • But I won’t be covering it in class. • So no, we can basically stop here. • So how can a sort be faster than mergesort? • Answer: the speedup is not asymptotic.

37. Quicksort • This is considered the fastest general purpose sort in the average case. • It is also a recursive “divide and conquer” sort. • Where mergesort simply split the array, Quicksortpartitions the array. • The difference is that a partition actually splits an unsorted array into 2 sorted arrays on each side. • Mergesort started with sorted arrays and built up. • Quicksort starts unsorted and splits down. • Base case: array of size 1.

38. History and Usage • This is probably the most popular sorting algorithm. • There are tons of variations. • Some are stable, some aren’t, some require extra space, some don’t, some parallelize well… • It is not the simplest, however. • Java uses it to sort arrays (but not collections). • It was developed by C.A.R. Hoare in 1962. • There is an issue that prevents most people from using the algorithm in its naïve form… • We’ll get to it.

39. Partitioning • Choose a pivot element from the array • Ideal is the median, but that slows the algorithm. • Most just use the first or last element. • Remove the pivot from the array. • Split the rest of the array into two: • Elements less than or equal to the pivot. • Elements greater than the pivot. • Concatenate the left array, the pivot, and the right array.

40. Partitioning algorithm. private static <Type extends Comparable<Type> > int partition(List<Type> l, int left, int right, intpivotidx) { //Temporarily move the pivot out of the way. swap(l, pivotidx, right); //Build the left list. intleftend = left; for (intcuridx = left; curidx < right; curidx++) if (l.get(curidx).compareTo(l.get(pivotidx)) <= 0) //Swap into left list. swap(l, curidx, leftend++); swap(l, leftend, right); //Move the pivot to the left end + 1. //The rest of the data is naturally on the right side now. return leftend; //Return the new position of the pivot. }

41. Quicksort • Once we partition, Quicksort is simple: //Quicksorts the collection type l. This is only efficient for random access structures. public static <Type extends Comparable<Type> > void quickSort(List<Type> l) { quickSort(l, 0, l.size()-1); } //Quicksorts the region of l between left and right in-place. private static <Type extends Comparable<Type> > void quickSort(List<Type> l, int left, int right) { if (right <= left) //Base case: size 1. return; intoldpivot = right; //Or any element. int pivot = partition(l, left, right, oldpivot); //The array is now partitioned around the pivot. quickSort(l, left, pivot - 1); //Elements <= pivot. quickSort(l, pivot + 1, right); //Elements > pivot. }

42. Quicksort Performance • On average, half of the elements will be less than the partition, half greater. • So we are usually splitting into two subproblems, each consisting of half the array. • Partitioning is linear. • This results in T(n) = 2T(n/2) + O(n). • Same as mergesort, same solution: O(n log n). • In practice, Quicksort is even faster than mergesort. • The loop in partition is easy for systems to optimize. • It also doesn’t require any extra space. • It isn’t stable.

43. Quicksort: the Achilles’ Heel. • Ah, but we’re not dealing with averages. We are interested in the worst case. • Everything ends up on one side of the pivot. • Well, that paints a bleaker picture: • Now we have a recurrence of T(n) = T(n-1) + O(n). • This is O(n^2). • The worst case occurs when the list is already sorted (if we choose a rightmost pivot). • Variations on Quicksort can get around this. • “Introsort” is a popular one.

44. Comparison • Bubble Sort, Insertion Sort, Selection Sort • O(n^2). • Bubble and insertion stable, selection not. • Little extra storage. • Primary advantage: simplicity. • Shell Sort: • Between O(n log n) and O(n^2). • Highly variable with gap selection. • Generalized insertion sort. • Not stable. • Primary advantage: fairly simple, fairly good performance. • Mergesort, Quicksort: • Optimal: O(n log n). • Mergesort either stable with O(n) memory usage or unstable in-place. • Quicksort generally unstable (stable variants exist). • Quicksort can be done in place or using extra memory. • Quicksort usually faster than Mergesort in practice. • But both algorithms are very good. • Primary disadvantage: Difficult to implement.

45. Putting things in order. • These are difficult topics. It’s ok to ask questions. • Now that we learned that O(n log n) was optimal, we’ll learn about faster sorts. • It’s the optimal comparison sort, you see. • The lesson: • Leverage existing work. All of the “good” algorithms are built into the language already. • Next class: Binary trees.