Understanding Decision Tree Lower Bounds

Lower bound: Decision tree and adversary argument • Lower bound: (f(n)) in the worst case. • Decision tree model: • Example: for comparison sorting. • Adversary argument: • Example: find the maximum

Comparison sort • Comparison sort: • Insertion sort, O(n2), upper bound in worst case • Merge sort, O(nlg n), upper bound in worst case • Heapsort, O(nlg n), upper bound in worst case • Quicksort, O(nlg n), in average case • Question: • what is the lower bounds for any comparison sorting: i.e., at least how many comparisons needed in worst case? • It turns out: Lower bound in worst case: (nlg n), how to prove? • Merge and Heapsort are asymptotically optimal comparison sorts.

Decision Tree Model • Assumptions: • All numbers are distinct (so no use for ai = aj ) • All comparisons have form ai aj (since ai aj, ai aj, ai< aj, ai> aj are equivalent). • Decision tree model • Full binary tree • Internal node represents a comparison. • Ignore control, movement, and all other operations, just see comparison • Each leaf represents one possible result. • The height (i.e., longest path) is the lower bound.

Decision tree model 1:2  > 2:3 1:3 >  >  <1,2,3> 1:3 <2,1,3> 2:3  >  > <1,3,2> <3,1,2> <2,3,1> <3,2,1> Internal node i:j indicates comparison between ai and aj. suppose three elements < a1, a2, a3> with instance <6,8,5> Leaf node <(1), (2), (3)> indicates ordering a(1) a(2) a(3). Path of bold lines indicates sorting path for <6,8,5>. There are total 3!=6 possible permutations (paths).

Lower bound: for comparison sort • The Longest path is the worst case number of comparisons. The length of the longest path is the height of the decision tree. • Theorem 8.1: Any comparison sort algorithm requires (nlg n) comparisons in the worst case. • Proof: • Suppose height of a decision tree is h, and number of paths (i,e,, permutations) is n!. • Since a binary tree of height h has at most 2h leaves, • n!  2h , so h  lg (n!)  (nlg n) (By equation 3.18). • That is to say: any comparison sort in the worst case needs at least nlg n comparisons.

Maximum: O(n) • MAXIMUM(A) • maxA[1] • fori 2 tolength[A] • doifmax<A[i] • thenmax A[i] • returnmax Running time: O(n), n-1 comparisons are sufficient. Similar for minimum.

Lower bound for maximum • n-1 is the upper bound of maximum, • How about the lower bound (of worst case)? • Suppose n elements are distinct. • So n-1 elements are not maximum. • Every comparison has only one loser. • Therefore at least n-1 comparisons are needed. • So lower bound is n-1. • This method is called the tournament method. • Can the decision tree is used for determining the lower bound for selection? • Any one could be the maximum, so there are n leaves (output) • Thus, the height will be lg n, • If think lg n be the lower bound, it will be wrong. • So decision tree does not apply here. • Why? ((really n leaves? Duplicate leaves!)

Adversary argument • Playing a guessing game between you and your friend. • You are to pick up a date, and the friend will try to guess the date by asking YES/NO questions. • Your purpose is forcing your friend to ask as many questions as possible. • To question “is it in winter”, your answer should be NO. • To question “is the first letter of the month’s name in the first half of the alphabet”? Your answer should be YES. • Idea: • You did not pick up a date in advance at all, but • Construct a date according to the questions. • The requirement is that the finally constructed date should be consistent to all your answers to the questions. • Looks like cheating, but it is a good way to find the lower bound.

Adversary argument • Suppose we have an algorithm we think efficient. • Image an adversary tries to prove otherwise. • At each point in the algorithm, whenever an decision (i.e., key comparison) is made, the adversary tells us the result of the decision. • The adversary chooses the answer which tries to force the algorithm work hard (i.e., do a lot of decision, or to say, the answer releases as less new information as possible). • You can think the adversary is constructing a “bad” input while it is answering the questions. The only requirement on the answers is that they must be internally consistent. • If the adversary can force the algorithm to perform f(n) steps, then f(n) is the lower bound, i.e, at least how many steps in the worst case.

Adversary argument for maximum • The adversary answers the questions as treating a[i]=i, for each i . • To the query “is a[i]<a[j]”, adversary answers YES if (and only if) i<j. • If the algorithm halts in less than n-1 comparisons and claims that the maximum is located in index k, then there will be one more non-loser (non-smaller element). Suppose the non-smaller element is located in the index j≠k • Thus, the adversary can thus demonstrate this algorithm to be incorrect by claiming that the array holds the values a[i] = i for all i≠j and a[j] = n+1, thus making a[k] < a[j]. A contradiction! • So the algorithm must have at least n-1 comparisons to determine the maximum. • So the previous linear scan algorithm for maximum is optimal. • If you design another algorithm, such as first divide into n/2 pairs and make comparisons to get winners, then divide the winners into pairs and make comparisons, …, the total number of comparisons is still n-1.

Adversary argument for finding both maximum and minimum. • n distinct elements, • Count each win and each lose as one unit of information. • One maximum and one minimum, so n-1 loses for maximum and n-1 wins for minimum, thus, at least 2n-2 units of information is needed. • The adversary gives the answer in the way that will give away as few as possible units of new information with each comparison.

Adversary argument—four status • Denote the key status in any moment as: • W: has won at least one comparison and never lost • L: has lost at least one comparison and never won • B: has both won and lost at least one comparison • N: has not yet participated in a comparison

Adversary argument --strategy Each W or L is one unit of info., B: 2 units of info. Except B,B, either key chosen as winner has not lost any comparisonor as loser has not won any comparison. Suppose the comparison is x and y, adversary chooses x as winner and x never loses. Even x value assigned previously is less than y, adversary can increases x to beat y without conflicting all previous answers.,

Adversary argument—proof • Except N,N case, all other cases release at most 1 unit of information. • So the algorithm compares two keys not involved in any comparison as much as possible: • Suppose n is even, then divide to n/2 pairs. thus obtain n units of information. There needs at least n-2 comparison, each for at most 1 unit of information, thus, n/2+n-2=3n/2-2. • If n is odd, (n-1)/2+n-1 =(3n-3)/2. • in one formula: 3n/2 -2.

Minimum and Maximum: 3n/2 comparisons • MIN-MAX(A) //assume n is odd • minmaxA[1] • fori 2 tolength[A] step 2 • doif A[i]<A[i+1] • thenifmin>A[i] • thenmin  A[i] • ifmax<A[i+1] • thenmax  A[i+1] • elseifmin> A[i+1] • thenmin  A[i+1] • ifmax<A[i] • thenmax  A[i] • returnmin, max #pairs: n/2 #comparisons/per pair:3 # total comparisons: 3n/2, i.e., 3n/2 -2. Similarly, write the code for n being even. The # total comparisons is 3n/2-2, i.e., 3n/2 -2. This algorithm achieves optimal.

Understanding Decision Tree Lower Bounds