Lower Bound Techniques for Data Structures

1. Lower Bound Techniquesfor Data Structures Mihai Pătrașcu … • Committee: • Erik Demaine (advisor) • PiotrIndyk • MikkelThorup

2. Data Structures I don’t study stacks, queues and binary search trees! I do study data structure problems (a.k.a. Abstract Data Types) partial-sums problem • Preprocess T = { n numbers } • pred(q): max { y єT | y < q} predecessor search • Maintain an array A[n] under: • update(i, Δ): A[i] = Δ • sum(i): return A[0] + … + A[i]

3. Motivation? • Support both:* list operations – concatenate, split, … • * array operations – index 0 1 2 3 0 1 2 3 4 • packet forwarding partial-sums problem • Preprocess T = { n numbers } • pred(q): max { y єT | y < q} predecessor search • Maintain an array A[n] under: • update(i, Δ): A[i] = Δ • sum(i): return A[0] + … + A[i]

4. Binary Search Trees = Upper Bound “Binary search trees solve predecessor search” => Complexity of predecessor ≤ O(lg n)/operation my work ≤ “Augmented binary search trees solve partial sums” => Complexity of partial sums ≤ O(lg n)/operation my work ≤ partial-sums problem • Preprocess T = { n numbers } • pred(q): max { y єT | y < q} predecessor search • Maintain an array A[n] under: • update(i, Δ): A[i] = Δ • sum(i): return A[0] + … + A[i]

5. What kind of “lower bound”? Lower bounds you can trust.TM Model of computation ≈ real computers: • memory words of w > lgn bits (pointers = words) • random access to memory • any operation on CPU registers (arithmetic, bitwise…) Just prove lower bound on # memory accesses “Array Mem[1..S] of w-bit words” “Black box”

6. Why Data Structures? I want to understand computation. • Other settings: • streaming L.B. : many not very “computational” mostly storage / info thy • space-bounded (PvsL) • L.B. : a few, Ω(n √lg n) unnatural questions • algebraic L.B. : some  cool, but not real computing… • depth 3 circuits with mod-6 gates ?? The gospel: • data structures L.B. : some understand some nontrivial computational phenomena • efficient algorithms circuit L.B. not forthcoming • hard optimization NP-completenessL.B. : one per STOC/FOCS 

7. Why Data Structures? I want to understand computation. • Other settings: • streaming L.B. : many not very “computational” mostly storage / info thy • space-bounded (PvsL) • L.B. : a few, Ω(n √lg n) unnatural questions • algebraic L.B. : some  cool, but not real computing… • depth 3 circuits with mod-6 gates ?? The gospel: • data structures L.B. : some understand some nontrivial computational phenomena • efficient algorithms circuit L.B. not forthcoming • hard optimization NP-completenessL.B. : one per STOC/FOCS  Weak as some of the lower bounds may be, it’s the area that has gotten farthest towards “understanding computation”

8. History* *Omitted: bounds for succinct data structures. • Observations: • huge influence • 2nd papers • result wrong (better upper bound known) • no journal version; many claims without proof [Yao, FOCS’78] [Ajtai’88] -- predecessor (static) [Fredman, Saks’89] -- partial sums, union find (dynamic)

9. History* *Omitted: bounds for succinct data structures. [Yao, FOCS’78] [Ajtai’88] -- predecessor (static) [Bing Xiao, Stanford’92] ** [Miltersen STOC’94] [Miltersen, Nisan, Safra, Wigderson STOC’95] [Beame, Fich STOC’99] [Sen ICALP’01] (1+ε)-nearest neighbor:• [Chakrabarti, Chazelle, Gum, Lvov STOC’99]• [Chakrabarti, Regev FOCS’04] [Fredman, Saks’89] -- partial sums, union find (dynamic) [Ben-Amram, Galil FOCS’91] [Miltersen, Subramanian, Vitter, Tamassia’93] [Husfeldt, Rauhe, Skyum’96] [Fredman, Henzinger’98]planar connectivity [Husfeldt, Rauhe ICALP’98]nondeterminism [Alstrup, Husfeldt, Rauhe FOCS’98]marked ancestor • [Alstrup, Husfeldt , Rauhe SODA’01]dynamic 2D NN [Alstrup, Ben-Amram, Rauhe STOC’99] union-find === richness lower bounds === [Borodin, Ostrovsky, Rabani STOC’99] p.m. [Barkol, Rabani STOC’00] rand. NN [Jayram,Khot,Kumar,Rabani STOC’03] p.m. [Liu’04] det. ANN

10. Three Main Ideas [Yao, FOCS’78] [Ajtai’88] -- predecessor (static) [Bing Xiao, Stanford’92] ** [Miltersen STOC’94] [Miltersen, Nisan, Safra, Wigderson STOC’95] [Beame, Fich STOC’99] [Sen ICALP’01] (1+ε)-nearest neighbor:• [Chakrabarti, Chazelle, Gum, Lvov STOC’99]• [Chakrabarti, Regev FOCS’04] [Fredman, Saks’89] -- partial sums, union find (dynamic) [Ben-Amram, Galil FOCS’91] [Miltersen, Subramanian, Vitter, Tamassia’93] [Husfeldt, Rauhe, Skyum’96] [Fredman, Henzinger’98]planar connectivity [Husfeldt, Rauhe ICALP’98]nondeterminism [Alstrup, Husfeldt, Rauhe FOCS’98]marked ancestor • [Alstrup, Husfeldt , Rauhe SODA’01]dynamic 2D NN [Alstrup, Ben-Amram, Rauhe STOC’99] union-find === richness lower bounds ===[Borodin, Ostrovsky, Rabani STOC’99] p.m. [Barkol, Rabani STOC’00] rand. NN [Jayram,Khot,Kumar,Rabani STOC’03] p.m. [Liu’04] det. ANN 3. Round Elimination 2. Asym. Communication, Rectangles 1. Epochs

11. Three Main Ideas [Yao, FOCS’78] [Ajtai’88] -- predecessor (static) [Bing Xiao, Stanford’92] ** [Miltersen STOC’94] [Miltersen, Nisan, Safra, Wigderson STOC’95] [Beame, Fich STOC’99] [Sen ICALP’01] (1+ε)-nearest neighbor:• [Chakrabarti, Chazelle, Gum, Lvov STOC’99]• [Chakrabarti, Regev FOCS’04] [Fredman, Saks’89] -- partial sums, union find (dynamic) [Ben-Amram, Galil FOCS’91] [Miltersen, Subramanian, Vitter, Tamassia’93] [Husfeldt, Rauhe, Skyum’96] [Fredman, Henzinger’98]planar connectivity [Husfeldt, Rauhe ICALP’98]nondeterminism [Alstrup, Husfeldt, Rauhe FOCS’98]marked ancestor • [Alstrup, Husfeldt , Rauhe SODA’01]dynamic 2D NN [Alstrup, Ben-Amram, Rauhe STOC’99] union-find === richness lower bounds ===[Borodin, Ostrovsky, Rabani STOC’99] p.m. [Barkol, Rabani STOC’00] rand. NN [Jayram,Khot,Kumar,Rabani STOC’03] p.m. [Liu’04] det. ANN 3. Round Elimination 2. Asym. Communication, Rectangles 1. Epochs

12. Review: Epoch Lower Bounds time update: mark/unmark node [tu] query: # marked ancestors? [tq] • epoch j: rj updates • epochs {0, .., j-1} write O(tuw∙rj-1) bits • pick r >>tuw most updates from epoch j not known outside epoch j random query needs to read a cell from epoch j max {tq, tu} = Ω(lg n / lglg n) tq= Ω(lg n / lg r) = Ω(lg n / lg(tuw))

13. Review: Epoch Lower Bounds • See also: • [FredmanJACM ’81] • [FredmanJACM ’82] • [Yao SICOMP ’85] • [Fredman, Saks STOC ’89] • [Ben-Amram, GalilFOCS ’91] • [Hampapuram, FredmanFOCS ’93] • [ChazelleSTOC ’95] • [Husfeldt, Rauhe, SkyumSWAT ’96] • [Husfeldt, RauheICALP ’98] • [Alstrup, Husfeldt, RauheFOCS ’98] “Big Challenges” [Miltersen’99] • prove some ω(lgn/lglgn) bound Candidate: Ω(lgn) for the partial sums problem • prove ω(lgn) in the bit-probe model Maintain an array A[n] under:update(i, Δ): A[i] += Δsum(i): return A[0] + … + A[i]

14. Our contribution [P., Demaine SODA’04]Ω(lgn) for partial sums [P., Demaine STOC’04]Ω(lgn) for dynamic trees, etc. * very simple proof * not based on epochs [P., Tarniţă ICALP’05]Ω(lgn) via epoch argument!! =>Ω(lg2n/lg2lg n) in the bit-probe model Best Student Paper

15. Ω(lgn) via Epoch Arguments? Old: information about epoch j outside j ≤ #cells written by epochs {0, .., j-1} ≤ O(tu∙rj-1) j

16. Ω(lgn) via Epoch Arguments? New: information about epoch j outside j ≤ #cells read by epochs {0, .., j-1} from epoch j still≤ O(tu∙rj-1) in the worst case  j • Foil worst-case by randomizing epoch construction!

17. Ω(lgn) via Epoch Arguments? #cells read by epochs {0, .., j-1} from epoch j ≤ O((tu/ #epochs) ∙ rj-1)on average => max { tu, tq } = Ω(lg n) • Foil worst-case by randomizing epoch construction!

18. The “Very Simple Ω(lg n) Proof”

19. π Maintain an array A[n] under: update(i, Δ): A[i] = Δ sum(i): return A[0] + … + A[i] Δ1 Δ2 The hard instance: π = random permutation for t = 1 to n:query: sum(π(t))Δt= rand()update(π(t), Δt) Δ3 Δ4 Δ5 Δ6 Δ7 Δ8 Δ9 Δ10 Δ11 Δ12 Δ13 Δ14 Δ15 Δ16 time

20. Δ1 Δ2 Δ3 Δ4 Δ5 Δ6 Δ7 Δ8 Δ9 Δ10 Δ11 Δ12 Δ13 • How can Mac help PC run ? Δ14 t = 9,…,12 Δ16 Δ17 Communication ≈ # memory locations * read during * written during time t = 9,…,12 t = 9,…,12 t = 5, …, 8 t = 5, …, 8

21. Δ1 Δ2 Δ3 Δ4 Δ5 Δ8 Δ7 Δ9 Δ1+Δ5+Δ3+Δ7+Δ2 Δ1 Δ1+Δ5+Δ3 Δ13 How much information needs to be transferred? Δ1+Δ5+Δ3+Δ7+Δ2+Δ8+Δ4 Δ14 Δ16 Δ17 time At least Δ5,Δ5+Δ7,Δ5+Δ7+Δ8 => i.e. at least 3 words (random values incompressible)

22. The general principle Lower bound = # down arrows How many down arrows? (in expectation) (2k-1) ∙ Pr[ ] ∙ Pr[ ] = (2k-1) ∙ ½ ∙ ½ = Ω(k) k operations k operations

23. Recap • Communication = # memory locations • * read during • * written during pink period yellow period • Communication between periods of k items = Ω(k) * read during * written during pink period # memory locations • = Ω(k) yellow period

24. Putting it all together aaaa • Ω(n/8) • Ω(n/4) Every load instruction counted once @ lowest_common_ancestor( , ) • Ω(n/8) • Ω(n/2) write time read time • Ω(n/8) • Ω(n/4) • Ω(n/8) • totalΩ(nlgn) time

25. Q.E.D. • Augmented binary search trees are optimal. • First “Ω(lgn)” for any dynamic data structure.

26. Three Main Ideas [Yao, FOCS’78] [Ajtai’88] -- predecessor (static) [Bing Xiao, Stanford’92] ** [Miltersen STOC’94] [Miltersen, Nisan, Safra, Wigderson STOC’95] [Beame, Fich STOC’99] [Sen ICALP’01] (1+ε)-nearest neighbor:• [Chakrabarti, Chazelle, Gum, Lvov STOC’99]• [Chakrabarti, Regev FOCS’04] [Fredman, Saks’89] -- partial sums, union find (dynamic) [Ben-Amram, Galil FOCS’91] [Miltersen, Subramanian, Vitter, Tamassia’93] [Husfeldt, Rauhe, Skyum’96] [Fredman, Henzinger’98]planar connectivity [Husfeldt, Rauhe ICALP’98]nondeterminism [Alstrup, Husfeldt, Rauhe FOCS’98]marked ancestor • [Alstrup, Husfeldt , Rauhe SODA’01]dynamic 2D NN [Alstrup, Ben-Amram, Rauhe STOC’99] union-find === richness lower bounds ===[Borodin, Ostrovsky, Rabani STOC’99] p.m. [Barkol, Rabani STOC’00] rand. NN [Jayram,Khot,Kumar,Rabani STOC’03] p.m. [Liu’04] det. ANN 3. Round Elimination 2. Asym. Communication, Rectangles 2. Asym. Communication, Rectangles 1. Epochs

27. Review: Communication Complexity

28. Review: Communication Complexity • lgS bits • w bits • lgS bits • w bits • database • => space S • query(a,b,c) Traditional communication complexity: “total #bits communicated ≥ X” =>tq∙(lg S + w) ≥ X => tq = Ω(X/w) But wait! X ≤ CPU input ≤ O(w)

29. Review: Communication Complexity • lgS bits • w bits • lgS bits • w bits • database • => space S • query(a,b,c) Asymmetric communication complexity: “either Alice sends A bits or Bob sends B bits” => either tq∙lg S ≥ A or tq∙w≥ B => tq≥ min { A/lg S, B/w}

30. Richness Lower Bounds Prove: “either Alice sends A bits or Bob sends B bits” Assume Alice sends o(A), Bob sends o(B) => big monochromatic rectangle Show any big rectangle is bichromatic (standard idea in comm. complex.) Bob output=1 1/2o(A) Alice 1/2o(B) Example: Alice --> q є {0,1}d Bob --> S=n points in {0,1}d Goal: find argminxєS|| x-q ||2 [Barkol, Rabani] A=Ω(d), B=Ω(n1-ε) => tq ≥ min { d/lg S, n1-ε/w }

31. Richness Lower Bounds • upper bound ≈ either: • exponential space • near-linear query time What does this really mean? “optimal space lower bound for constant query time” tq n1-o(1) Θ(d/lg n) 1 S lower bound S = 2Ω(d/tq) Θ(n) 2Θ(d) Example: Alice --> q є {0,1}d Bob --> S=n points in {0,1}d Goal: find argminxєS|| x-q ||2 [Barkol, Rabani] A=Ω(d), B=Ω(n1-ε) => tq ≥ min { d/lg S, n1-ε/w } Also: optimal lower bound for decision trees

32. Results Partial match -- database of n strings in {0,1}d, query є {0,1,*}d[Borodin, Ostrovsky, Rabani STOC’99][Jayram,Khot,Kumar,Rabani STOC’03] A=Ω(d/lg n) [P. FOCS’08]A = Ω(d) Nearest Neighbor on hypercube (ℓ1, ℓ2): deterministicγ-approximate: [Liu’04] A = Ω(d/γ2) randomized exact: [Barkol, Rabani STOC’00]A = Ω(d) rand. (1+ε)-approx: [Andoni, Indyk, P. FOCS’06] A = Ω(ε-2lg n) “Johnson-Lindenstrauss space is optimal!” Approximate Nearest Neighbor in ℓ∞: [Andoni, Croitoru, P. FOCS’08]“[Indyk FOCS’98] is optimal!” simplify

33. Limits of Communication Approach tq • lgS bits branchingprograms n1-o(1) Θ(d/lg d) • w bits Implication of richness lower bound undervalued! Θ(d/lg n) 1 S Θ(n) 2Θ(d) “ Alice must send Ω(A) bits” => tq= Ω(A / lg S) => tq= Ω(A / lg(Sd/n)) No separation between S=O(n) and S=nO(1)! Separation of Ω(lgn / lglgn) betweenS=O(n)and S=nO(1)!

34. Richness Gets You More • CPU(s) -->memory communication: • one query: lgS • kqueries: lg( )=Θ(klg) S k S k

35. Richness Gets You More • CPU(s) -->memory communication: • one query: lgS • kqueries: lg( )=Θ(klg) S k S k Prob.1 Prob.3 Prob.2 Prob.k

36. Richness Gets You More • CPU(s) -->memory communication: • one query: lgS • kqueries: lg( )=Θ(klg) S k S k Prob.1 Prob.3 Prob.2 Direct Sum Prob.k Any richness lower bound “Alice must send A or Bob must send B” ===> k∙Alicemust send k∙A or k∙Bobmust send k∙B

37. Richness Gets You More • CPU(s) -->memory communication: • one query: lgS • kqueries: lg( )=Θ(klg) tq= Ω(A / lg(S/k)) S k S k Direct Sum Any richness lower bound “Alice must send A or Bob must send B” ===> k∙Alicemust send k∙A or k∙Bobmust send k∙B

38. Three Main Ideas [Yao, FOCS’78] [Ajtai’88] -- predecessor [Bing Xiao, Stanford’92] [Miltersen STOC’94][Miltersen, Nisan, Safra, Wigderson STOC’95] [Beame, Fich STOC’99] [Sen ICALP’01] (1+ε)-nearest neighbor: • [Chakrabarti, Chazelle, Gum, Lvov STOC’99] • [Chakrabarti, Regev FOCS’04] [Fredman, Saks’89] - partial sums, union find [Ben-Amram, Galil FOCS’91] [Miltersen, Subramanian, Vitter, Tamassia’93] [Husfeldt, Rauhe, Skyum’96][Fredman, Henzinger’98]planar connectivity [Husfeldt, Rauhe ICALP’98]nondeterminism [Alstrup, Husfeldt, Rauhe FOCS’98]marked ancestor [Alstrup, Husfeldt , Rauhe SODA’01] dynamic 2D NN [Alstrup, Ben-Amram, Rauhe STOC’99] union-find === richness lower bounds ===[Borodin, Ostrovsky, Rabani STOC’99] p.m. [Barkol, Rabani STOC’00] rand. NN [Jayram,Khot,Kumar,Rabani STOC’03] p.m. [Liu’04] det. ANN 3. Round Elimination 2. Asym. Communication, Rectangles 1. Epochs 4. Range Queries

39. Open Hunting Season Nice trick, but “Ω(lgn / lglgn) with O(npolylgn) space” not impressive argument for “curse of dimensionality” But space n1+o(1) is hugely important in data structures => open hunting season for range queries etc. 71000 2D range counting 70000 69000 • SELECT count(*) • FROM employees • WHERE salary <= 70000 • AND startdate <= 1998 68000

40. Open Hunting Season • [P. STOC’07] Ω(lgn / lglgn) with O(npolylgn) space N.B. : tight! • 1st bound beyond the semigroup model question from [FredmanJACM’82] [ChazelleFOCS’86] 71000 2D range counting 70000 69000 • SELECT count(*) • FROM employees • WHERE salary <= 70000 • AND startdate <= 1998 68000

41. The Power of Reductions 2D stabbing • Preprocess S={n rectangles} • • stab(x,y): is (x,y) inside some RєS? routing ACLs dispatching in some OO languages 71000 2D range counting 70000 69000 • SELECT count(*) • FROM employees • WHERE salary <= 70000 • AND startdate <= 1998 68000

42. The Power of Reductions -1 +1 -1 +1 2D stabbing +1 -1 -1 +1 -1 • Preprocess S={n rectangles} • • stab(x,y): is (x,y) inside some RєS? +1 -1 +1 +1 -1 +1 -1 71000 2D range counting 70000 69000 • SELECT count(*) • FROM employees • WHERE salary <= 70000 • AND startdate <= 1998 68000

43. The Power of Reductions 2D stabbing • Preprocess S={n rectangles} • • stab(x,y): is (x,y) inside some RєS? reachability oracles in butterfly graph • Preprocess G = subgraph of butterfly • • reachable(x,y): is there a path x->y ?

44. The Power of Reductions 2D stabbing • Preprocess S={n rectangles} • • stab(x,y): is (x,y) inside some RєS? reachability oracles in butterfly graph • Preprocess G = subgraph of butterfly • • reachable(x,y): is there a path x->y ?

45. The Power of Reductions Lopsided Set Disjointness • Alice: set SBob: set T“are S and T disjoint?” Hint: S = {one edge out of every node} => n queries from 1st to last level T = {deleted edges}S disjoint from T => all queries “yes” reachability oracles in butterfly graph • Preprocess G = subgraph of butterfly • • reachable(x,y): is there a path x->y ?

46. Reachability in Butterfly?? marked ancestor problem update(node): (un)mark node query(leaf):any marked ancestor?

47. lopsided set disjointness (LSD) reachability oracles in the butterfly partial match (1+ε)-ANN ℓ1, ℓ2 NN in ℓ1, ℓ2 dyn. marked ancestor 3-ANN in ℓ∞ 2D stabbing worst-case union-find dyn. trees, graphs 4D reporting 2D counting dyn. 1D stabbing [P. FOCS’08] partial sums dyn. 2D reporting dyn. NN in 2D

48. Three Main Ideas [Yao, FOCS’78] [Ajtai’88] -- predecessor [Bing Xiao, Stanford’92] [Miltersen STOC’94][Miltersen, Nisan, Safra, Wigderson STOC’95] [Beame, Fich STOC’99] [Sen ICALP’01] (1+ε)-nearest neighbor: • [Chakrabarti, Chazelle, Gum, Lvov STOC’99] • [Chakrabarti, Regev FOCS’04] [Fredman, Saks’89] - partial sums, union find [Ben-Amram, Galil FOCS’91] [Miltersen, Subramanian, Vitter, Tamassia’93] [Husfeldt, Rauhe, Skyum’96][Fredman, Henzinger’98]planar connectivity [Husfeldt, Rauhe ICALP’98]nondeterminism [Alstrup, Husfeldt, Rauhe FOCS’98]marked ancestor [Alstrup, Husfeldt , Rauhe SODA’01] dynamic 2D NN [Alstrup, Ben-Amram, Rauhe STOC’99] union-find === richness lower bounds ===[Borodin, Ostrovsky, Rabani STOC’99] p.m. [Barkol, Rabani STOC’00] rand. NN [Jayram,Khot,Kumar,Rabani STOC’03] p.m. [Liu’04] det. ANN 3. Round Elimination 3. Round Elimination 2. Asym. Communication, Rectangles 1. Epochs 4. Range Queries

49. Packet Forwarding/ Predecessor Search Preprocess n prefixes of ≤ w bits:  make a hash-table H with all prefixes of prefixes  |H|=O(n∙w), can be reduced to O(n) Given w-bit IP, find longest matching prefix:  binary search for longest ℓ such that IP[0: ℓ] єH [van Emde Boas FOCS’75] [Waldvogel, Varghese, Turener, PlattnerSIGCOMM’97] [Degermark, Brodnik, Carlsson, Pink SIGCOMM’97] [Afek, Bremler-Barr, Har-PeledSIGCOMM’99] O(lgw)

50. Review: Round Elimination hi lo hash(hi) 0/1† I want to talk to Alice † 0: continue searching for pred(hi) 1: continue searching for pred(lo) i 1 o(k) bits Message has negligible info about the typical i => can be eliminated for fixed i 2 k