Ch. 8 & 9 – Linear Sorting and Order Statistics

Ch. 8 & 9 – Linear SortingandOrder Statistics What do you trade for speed?

Ch.8 – Linear Sorting Sorting by Comparisons Up to this point, all the sorting algorithms we examined depended on comparing elements of a given set, in order to sort the set. All the algorithms we came up with, were either O(n lg n) or Q(n lg n) or O(n2). One can ask: can we sort a set S, consisting of elements from a totally ordered universe, in time O(|S|)? The answer, as we might expect, is “yes, but…” First of all, the negative result: sorting by comparisons is (worst case) W(n lg n). 91.404

Ch.8 – Linear Sorting .. 91.404

Ch.8 – Linear Sorting The main ideas are: • Every time you try to determine the relative positions of two element you must make a comparison (decision). • The input (n elements) can come in any order. • There are n! ways in which n different elements can be arranged. • A “sort” is equivalent to finding (by a sequence of comparisons between two elements) the permutation of the input that leaves the input set ordered. • Each such permutation corresponds to a leaf of the “binary decision tree” generated by the comparisons. • The binary decision tree has n! leaves. 91.404

Ch.8 – Linear Sorting Theorem 8.1: Any comparison sort algorithm requires W(n lg n) comparisons in the worst case. Proof: by the previous discussion, the binary decision tree has at least n! leaves, and height h. Since such a binary tree cannot have more than 2h leaves, we have n! ≤ 2h. Taking the logarithm (base 2) of both sides: h ≥ lg (n!) = W(n lg n), This means that there is at least ONE path of length h connecting the root to a leaf. Corollary 8.2: HEAPSORT and MERGESORT are asymptotically optimal comparison sorts. 91.404

Ch.8 – Linear Sorting Sorting NOT by comparisons How do we do it? • We must make some further assumptions. • For example, we need to assume more than “the set to be sorted is a set of integers”. • More specifically, we assume the integers fall in some range, say [1..k], where k = O(n). • This is the idea behind Counting Sort. How do we use it? 91.404

Ch.8 – Linear Sorting .. 91.404

Ch.8 – Linear Sorting Counting Sort: Time Complexity How much time? • The for loop of l. 2-3 takes time Q(k). • The for loop of l. 4-5 takes timeQ(n). • The for loop of l. 7-8 takes time Q(k). • The for loop of l. 10-12 takes time Q(n). • The overall time is Q(k + n). • The assumption on k gives that the overall time is Q(n). 91.404

Ch.8 – Linear Sorting Radix Sort – or Hollerith’s Sort http://www.columbia.edu/acis/history/hollerith.html What else can we use? Assume all your integers (we are sorting sets of integers) have d or fewer digits. Pad the ones with fewer than d digits with leading 0s, for uniformity. Assume the digits are on cards with 80 or so vertical columns, each column with room for 10 (or more) distinct holes (one for each of 0..9). Use columns 1..d to store each integer. Take a deck of such cards (with integers) and sort it. 91.404

Ch.8 – Linear Sorting Radix Sort 91.404

Ch.8 – Linear Sorting Radix Sort Lemma 8.3. Given nd-digit numbers in which each digit can take up to k possible values, RadixSort correctly sorts these numbers in Q(d(n + k)) time if the stable sort it uses takes Q(n + k) time. Proof: the correctness follows by induction on the column being sorted. Sort the first column (using a stable sort). Assume the set is sorted on the first i columns (starting from the back); prove it remains sorted when we use a stable sort on the i+1stcolumn (see Ex. 8.3-3). CountingSort on each digit will give the result (details?). 91.404

Ch.8 – Linear Sorting Bucket Sort Assume all the values have equal (independent) probability of appearing as elements of [0, 1). • Divide the interval [0, 1) into n equal sized subintervals (buckets) and distribute the n input numbers into the buckets. • Sort the numbers in each bucket (your choice of sort). • Go through the buckets in order, listing the contents. 91.404

Ch.8 – Linear Sorting Bucket Sort 91.404

Ch.8 – Linear Sorting Bucket Sort We observe that both of the for loops in l. 3-4 and 5-6 take time O(n) to execute. We need to analyze the cost of the n calls to InsertionSorton l. 8. 91.404

Ch.8 – Linear Sorting Bucket Sort Let ni be the random variable denoting the number of elements in bucket B[i]. Since INSERTIONSORT runs in quadratic time, the running time for BUCKETSORT is The expected time is given by We will show that E[ni2] = 2 – 1/n, fori = 0, 1, ..., n - 1. It should be clear that the expected number of items in each bucket is the same: by the uniform distribution. 91.404

Ch.8 – Linear Sorting Bucket Sort Define the indicator random variables Xij = I{A[j] falls in bucket i}, for i = 0, …, n - 1,j = 1, …, n. Thus To compute E[ni2], we expand, square and regroup: 91.404

Ch.8 – Linear Sorting Bucket Sort We now compute the two sums. The indicator random variable Xij takes the value 1 with probability 1/n and 0 with probability 1 – 1/n. The same is true of Xij2: E[Xij2] = 12*(1/n) + 02*(1 – 1/n) = 1/n. We observe that, with k ≠ j, Xij and Xik are independent. This leads to: E[XijXik] = E[Xij]*E[Xik] = (1/n)*(1/n) = 1/n2. 91.404

Ch.8 – Linear Sorting Bucket Sort Substitute in the last equation two slides ago: It follows that E[T(n)] = Q(n) + n*O(2 – 1/n) = Q(n). Note: the same result holds as long as the distribution implies that the sum of the bucket sizes is linear in the total number of elements. 91.404

Ch.9 – Order Statistics Question: what is the time complexity for the extraction of the ith element in a totally ordered set of n elements?Answer: we first have to find an algorithm… We can find a minimum or a maximum in (deterministic) time Q(n) - how? To find the ith element, we can partition the set into two parts (as in QUICKSORT), if the pivot is the ith element, we are done; if not, we can decide which side of the pivot will contain the ith element and partition that subset further. 91.404

Ch.9 – Order Statistics The algorithm: we use randomization to offset the probability of receiving the worst possible input sequence. 91.404

Ch.9 – Order Statistics Note that the recursive call is only on one side of the partition. That is what will allow us to conclude an expected time complexity that is linear in n. Obviously, the worst case is Q(n2). • Let T(n) be the random variable that denotes the running time of the algorithm on an input array A[p..r] of n elements. We obtain an upper bound on E[T(n)]. • RANDOMIZED-PARTITION is equally likely to return any element as the pivot. • Therefore, for each kÎ[1, …, n], the subarray A[p..q] has k elements (all ≤ the pivot) with probability 1/n. 91.404

Ch.9 – Order Statistics • For kÎ[1, …, n], define the indicator random variable Xk = I{the subarray A[p..q] has exactly k elements}. • If the elements of A are distinct, E[Xk] = 1/n. • When we call RANDOMIZE-SELECT and choose A[q] as the pivot element, we do not know a priori, if we will terminate immediately, recurse on the left subarray A[p, …, q-1] or recurse on the right subarray A[q+1, …, r]. • The decision depends on where the ith smallest element falls w.r.t. A[q]. Since we are looking for an upper bound, we will assume the desired element is always in the larger of the two partition subarrays – and we’ll assume T(n) to be monotone increasing. 91.404

Ch.9 – Order Statistics • For a given call of Randomized-Select, the indicator random variable Xk has the value 1 for exactly one value of k, and has the value 0 for all oher values of k. • When Xk= 1, the two subarrays available for the recursion have sizes k - 1 and n - k. • The recurrence is • Taking Expected values (Xk & T(max(k-1, n-k)) indep.): 91.404

Ch.9 – Order Statistics • We now need to simplify the last expression: observe that • If n is even, each term from T(ceil(n/2)) up to T(n-1) appears exactly twice in the summation; • If n is odd, each term from T(ceil(n/2)) up to T(n-1) appears exactly twice in the summation along with one appearance of T(floor(n/2)). This leads to the estimate: The rest of the proof uses this estimate and induction on the substitution E[T(n)] ≤ cn, for some c > 0 and large enough n. 91.404

Ch.9 – Order Statistics Note: with a more complex algorithm, one can compute order statistics in deterministic time Q(n). See the text. 91.404

Ch. 8 & 9 – Linear Sorting and Order Statistics