Programming, Data Structures and Algorithms (Sorting)

1. Programming, Data Structures and Algorithms (Sorting) Anton Biasizzo

2. Preliminaries • Problem of sorting an array of elements. • We will assume that array contains only integers. • We will assume that N is the number of elements passed to the sorting algorithm. • For some algorithms it is convenient to place a sentinel in position 0, thus array will range from 0 to N. • Actual data starts at position 1. • Only allowed operations on the input data are comparisons and assignment. • Comparison-based sorting.

3. Insertion sort • Insertion sort is one of the simplest algorithms. • It consists of N-1 passes. • For pass p=2 through N, insertion sort ensures that the elements in positions 1 through p are in sorted order. • It makes use of the fact that elements in positions 1 through p-1 are already in sorted order. • Algorithm: • In pass p we move p-th element left until its correct place is found among the first p elements. • The sentinel in a[0] terminates the loop in the event that in some pass an element is moved all the way to the front.

4. Insertion sort • Insertion sort: void insertion_sort(input_type a[], unsigned int n) { unsigned int j, p; input_type tmp; a[0] = MIN_DATA; for ( p=2; p<=n; p++) { tmp = a[p]; for ( j=p; tmp<a[j-1]; j-- ) a[j] = a[j-1]; a[j] = tmp; } }

5. Insertion Sort analysis • Because of nested loops, each of which can take N iterations, insertion sort is O(N2). • Comparison in the inner for loop is executed at most • This bound is tight because input in reverse order actually achieve that bound. • If the input is pre-sorted, the running time is O(N) because the test of the inner for loop fails immediately. • Running time of an average case for insertion sort is O(N2).

6. Lower bound for simple sorting algorithm • An inversion in an array of numbers is any ordered pair (i,j) having the property that i<j but a[i]>a[j]. • Number of inversions is the number of swaps that needed to be performed in the insertion sort. • Sorted array has no inversions. • Swapping two adjacent elements that are out of place removes exactly one inversion. • In the insertion sort there is O(N) other work involved in algorithm. • The running time of insertion sort is O(I+N), where I is the number of inversions in the original array. • Insertion sort runs in linear time if the number of inversions is N.

7. Lower bound for simple sorting algorithm • The average number of inversions in an array of N distinct numbers is N (N - 1)/4. • Proof: • For any list, L, of numbers, consider Lr, the list in reverse order. • Consider any pair of two numbers in the list (i,j), with j>i. Clearly, in exactly one of L and Lr this ordered pair represents an inversions. • The total number of these pairs in L and Lr is N(N-1)/2. • An average list has half of this amount: N (N-1)/4. • Insertion sort is quadratic on average, • Any algorithm that sorts by exchanging adjacent elements requires Ω(N2) time on average. • Proof: • Initially, the average number of inversions is N (N-1) / 4 = Ω(N2). • Each swap removes only one inversion, so Ω(N2) swaps are required.

8. Shellsort • Named by inventor Donald Shell. • Breaks the quadratic barrier: • Compare distant elements, • The distance is decreased in each phase, • In the last phase adjacent elements are compared. • Shellsort uses a sequence, h1, h2, …, ht, called the increment sequence. • Any sequence is suitable as long as ht =1. • The algorithm performance depends on the sequence. • In a k-th phase, using increment hk, all elements spaced hk apart are sorted. • After a phase, using increment hk, for every i, we have A[i] ≤ A[i+ hk] and array is said to be hk-sorted. • An hk-sorted array that is then hk+1-sorted remains hk-sorted.

9. Shellsort • Strategy for hk-sort: • For each position i, in hk+1, hk+2,…,N, place the element in the correct spot among i, i-hk, i-2hk, … • An hk-sort performs an insertion sort on hk independent sub-arrays. • A common (but poor) choice of increment sequence is h1=N/2, hk+1=hk /2 (Shell’s increments). • Increment sequences significantly affect the performance of shellsort algorithm.

10. Analysis of Shellsort • Average-case analysis of Shellsort is a long-standing open problem (except for most trivial increment sequences). • The worst-case running time, using Shell’s increments, is θ(N2). Proof: • Determine lower Ω(N2) and upper bound O(N2). • Determine lower bound by constructing a bad case: • Let N be a power of 2 – all the increments are even except h1, which is 1. • Let the N/2 largest elements be in even positions and N/2 smallest elements be in odd positions. • Since all increments except the last are even, the N/2 largest elements are still all in even positions, and N/2 smallest elements are in odd positions, when we come to the last pass. • The ith smallest element (i ≤ N/2) is in position 2i-1 before beginning the last pass and requires moving for i-1 places. • To place the N/2 smallest elements in the correct place require

11. Analysis of Shellsort • Determine upper bound: • A pass with increment hk consists of hk insertion sorts of N/hk elements. • Since insertion sort is quadratic, the total cost of the pass is O(hk (N/hk)2) = O(N2/hk) • Summing over all passes gives a total bound • Increments form a geometric series with common ratio 2, hence the • Obtained total bound is O(N2).

12. Analysis of Shellsort • Problem with Shell’s increments: they are not necessarily relatively prime and some increments can have little effect. • Hibbard suggested a slightly different increment sequence of form 1, 3, 7, …, 2k-1. • Key difference: the consecutive increments have no common factors. • The worst-case running time of Shellsort, using Hibbard’s increments, is θ(N3/2). • Based on simulations, the average-case running time of Shellsort, using Hibbard’s increments, is thought to be O(N5/4). • Sedgewick proposed several increments sequences that give an O(N4/3) worst-case running time, the average running time is conjectured to be O(N7/6). • One of the sequences is {1, 5, 19, 41, 109, …}, in which terms are either of form 9·4i – 9·2i + 1 or 4i – 3·2i +1. • These sequences are most easily implemented by placing its values in an array.

13. Heapsort • Priority queues can be used to sort in O(N log N) time. • Heapsort gives the best running time growth rate so far. • In practice it is slower than a Shellsort with Sedgewick’s increment sequence. • Sort strategy: • Build a binary heap of N elements – it takes O(N) time. • Perform N DeleteMin operations, the elements leave the heap smallest first, in sorted order. • By recording these elements in a second array and copying the array back we sort N elements. • Each DeleteMin requires O(log N) time. • The total running time is O(N log N).

14. Heapsort • The clever way to avoid additional array is to use the fact, that after each DeleteMin, the heap shrinks by 1 thus the cell that was last in the heap can store just deleted (minimum) element. • This way we got array sorted in decreasing order. • Change the ordering property of the heap, so that the parent has a larger key than the child (max Heap).

15. Analysis of Heapsort • Building the heap uses at most 2N comparisions. • In second phase, the ith DeleteMax uses at most 2 log i comparisons, for total at most 2N log N – O(N) comparisons. • Consequently, in the worst case, at most 2N log N – O(N) comparisons are used by heapsort. • Experiments have shown that heapsort is extremely stable algorithm: on average it uses only slightly fewer comparisons than the worst case bound suggest. • Heapsort’s average running time is hard to estimate because successive DeleteMax operations destroy the heap’s randomness.

16. Mergesort • Mergesort runs in in O(N log N) worst-case running time. • It is good example of a recursive algorithm. • The fundamental operation is merging two sorted lists. Because the lists are sorted this can be done in one pass. • The basic merging algorithm takes: • two input arrays A and B, and output array C. • three counters (aptr, bptr, and cptr) for the corresponding arrays which are initially set to the beginning of the arrays. • The smaller of A[aptr] and B[bptr] is copied to the next entry in C[cptr] and appropriate counters are advanced.

17. Mergesort • When one array is exhausted, • the remainder of the other array is copied to output array

18. Mergesort • The time to merge two sorted lists is clearly linear, because at most N-1 comparisons are made. • Every comparison adds an element to set C, except the last one, which adds two. • The mergesort algorithm is easy to describe: • If N=1 the is one element to sort and the array is already sorted. • Otherwise recursively mergesort first and second half of the array. • This algorithm presents classic divide and conquer strategy. • The problem is divided into smaller problems and solved recursively. • The conquer phase consist of patching together the answers. • If temporary array is declared locally, then there could be log N temporary arrays active – dangerous on machine with small memory. • Only one temporary array is needed. • It can be rewritten without recursion.

19. Analysis of Mergesort • Let us assume that N is a power of 2. • For N=1 the time to mergesort is constant and denoted by 1. • The time to mergesort N elements is equal to time to do two recursive mergesorts of size N/2, plus the time to merge, which is linear: • T(1) = 1 • T(N) = 2 T(N/2) + N • Dividing second equation by N we get • And finally

20. Analysis of Mergesort • After everything is added we get because there is log N equations. • T(N) = N log N + N = O(N log N) • Although mergesort’s running time is O(N log N), it is rarely used for main memory sorts. • It needs linear extra memory and additional work spent for copying the temporary array slows down the algorithm considerably.

21. Quicksort • The quicksort is the fastest known sorting algorithm in practice. • Its average running time is O(N log N). • It has O(N2) worst-case performance, but it can be made very unlikely with little effort. • Like mergesort, quicksort is divide and conquer recursive algorithm. • The basic steps to sort an array S are: • If the number of elements in S is 0 or 1, then return. • Pick an element v in S. It is called the pivot. • Partition S-{v} into two disjoint groups: S1 is a set of elements that are smaller or equal to the pivot and S2 is a set of elements that are bigger or equal to the pivot. • Return { quicksort(S1), v, quicksort(S2) }.

22. Quicksort

23. Quicksort • The algorithm is ambiguous about what to do with the elements equal to the pivot – this becomes a design decision. • Why is quicksort better then mergesort? • Like mergesort, it recursively solves two sub-problems and requires linear additional work. • The sub-problems are not guaranteed to be of equal size (potentially bad). • Partitioning step can be performed in place and very efficient.

24. Picking the Pivot • A wrong way: • The popular choice is to use the first element as the pivot. • This is acceptable if the input is random. • If the input is pre-sorted or in the reverse order than all the elements go into S1 or S2, throughout the recursive calls. In such case it takes quadratic time. • This is quite frequent. • A Safe Manoeuvre • Pick the pivot randomly. • Depend on random number generator (might be poor). • Random number generator is expensive. • Median of Three Partitioning • The median of a group of N numbers is the N/2-th largest number. • Best choice is the median of complete file but is hard to calculate. • A good estimate is a median of three elements: left, right, and center elements.

25. Partitioning strategy • There are several partitioning strategies in practice. • Partitioning strategy, which yields good results: • Get the pivot out of the way by swapping it with last element. • i starts at the first element and j starts at the next-to-last element • While i is left to j, we move i right skipping over elements smaller than the pivot. We move j left, skipping over elements that are larger than the pivot. • When i and j have stopped, i is pointing at a large element and j is pointing at a small element. • If i is left of j, those elements are swapped.

26. Partitioning strategy • How to handle elements that are equal to the pivot: • If both pointers stop there are extra swapping, but they will cross in the middle. • If pointers skip equal elements, no swap is performed, however the pivot is put to the next-to-last position, which gives O(N2) running time if all elements are equal.

27. Partitioning strategy • For small files quicksort does not perform as well as insertion sort (N≤20). • Because quicksort is recursive these cases will occur frequently. • Use other sorting algorithm for small files. • If the pivot is median of left, right, and center element, we can order these elements and hide pivot. • Left and right element perform as a sentinel • The starting point of both pointers can be moved by one.

28. Analysis of Quicksort • Like mergesort quicksort is recursive • T(0)=T(1)=1 • The running time of quicksort is equal to running time of the two recursive calls plus linear time spent in partition: • T(N) = T(i) + T(N-i-1) + cN • Worst-case analysis (pivot is smallest element; i=0, we ignore T(0)=1): • T(N) = T(N-1) + cN, N>1

29. Analysis of Quicksort • Best-Case analysis: • The pivot is in the middle • T(N) = 2 T(N/2) + cN, N>1 • We add all equations and get • Which yields

30. Selection Problem • Selection problem: Find k-th largest (smallest) element • Using priority queue, we can find it in O(N+k logN). Finding median requires O(N logN) time. • Since we can sort the array in O(N logN) time we expect to obtain better time bound for selection. • Quick-select algorithm (|S| denotes number of elements in S): • If |S| = 1 (only when k=1), return the element of S as answer. • Pick a pivot element v. • Partition S - {v} into S1 and S2, as in quicksort. • If k ≤ |S1| then the k-th element is in S1 and return quickselect(S1, k) • If k = 1+|S1| then the pivot is the k-th smallest element. • Otherwise, the k-th smallest element is in S2 and return quickselect(S2, k-|S1|-1). • The worst-case running time is O(N2), however average-case running time is O(N).

31. A General Lower Bound for Sorting • We have O(N log N) algorithms for sorting, but can we do better? • Algorithm that uses only comparisons requires Ω(N log N) comparisons in the worst case. • The same bound can be proven for average case. • Decision tree is an abstraction used to prove lower bounds: • It is a binary tree. • Each node represents a set of possible orderings, consistent with the comparisons that have been made, among the elements. • The results of the comparisons are the tree edges.

32. A General Lower Bound for Sorting • A decision tree for three element insertion sort:

33. A General Lower Bound for Sorting • Every algorithm that sorts by using only the comparisons can be represented by a decision tree. • The maximum number of comparisons used by the algorithm is equal to the depth of the deepest leaf. • In our case this algorithm uses three comparisons in the worst case. • The average number of comparisons used is equal to the average depth of the leaves. • Lemma 1: Let T be a binary tree of depth d. Then T has at most 2d leaves. • Proof: By induction: • If d=0, the there is at most one leaf. • Otherwise, we have root, which cannot be a leaf, and left and right subtree, each of depth at most d-1. By induction hypothesis they can each have at most 2d-1 leaves, giving a total of 2d leaves.

34. A General Lower Bound for Sorting • Lemma 2: A binary tree with L leaves must have depth at least [log L]. • Proof: Follows from previous lemma. • Theorem 1: Any sorting algorithm that uses only comparisons between elements requires at least [log (N!)] comparisons in the worst case. • Proof: A decision tree to sort N elements have N! leaves. The theorem follows from previous lemma. • Theorem 2: Any sorting algorithm that uses only comparisons between elements requires Ω(N log N) comparisons.

35. A General Lower Bound for Sorting • Proof: From previous theorem, log(N!) comparisons is required. log(N!) = log( N (N-1) (N-2) ··· 2 1) = log N + log(N-1) + log (N-2) + ··· + log 2 + log 1 ≥ log N + log(N-1) + log (N-2) + ··· + log (N/2) ≥ N/2 log (N/2) ≥ N/2 log N – N/2 = Ω( N log N)

36. Bucket Sort • In special cases it is possible to sort in linear time. • The input consists of only positive integers smaller than M. • Algorithm: • Array Count, of size M, is initialized to 0, (Count has M cells or buckets, which are initially empty), • When Ai is read, increment Count[Ai] by 1, • After all the input is read, scan the Count array and printout indexes of non-empty cells. • Algorithm violates the lower bound? • It does not use simple comparisons. • Essentially performs M-way comparison.