Information Complexity Lower Bounds

1. Information Complexity Lower Bounds Rotem Oshman, Princeton CCI Based on: Bar-Yossef,Jayram,Kumar,Srinivasan’04 Braverman,Barak,Chen,Rao’10

2. Communication Complexity = ? Yao ‘79, “Some complexity questions related to distributive computing”

3. Communication Complexity • Applications: • Circuit complexity • Streaming algorithms • Data structures • Distributed computing • Property testing • …

4. Deterministic Protocols • A protocol specifies, at each point: • Which player speaks next • What should the player say • When to halt and what to output • Formally, what we’ve said so far who speaks next: Alice, Bob, = halt what to say/output

5. Randomized Protocols • Can use randomness to decide what to say • Private randomness: each player has a separate source of random bits • Public randomness: both players can use the same random bits • Goal: for any compute correctly with probability • Communication complexity: worst-case length of transcript in any execution

6. Randomness Can Help a Lot • Example: Equality • Input: • Output: is ? • Trivial protocol: Alice sends to Bob • For deterministic protocols, this is optimal!

7. Equality Lower Bound #rectangles

8. Randomized Protocol • Protocol with public randomness: • Select random • Alice sends • Bob accepts iff • If : always accept • If : • Reject with probability non-zero vector

9. Set Disjointness • Input: • Output: ? • Theorem [Kalyanasundaran, Schnitger ‘92, Razborov ‘92]: randomized CC = • Easy to see for deterministic protocols • Today we’ll see a proof by Bar-Yossef, Jayram, Kumar, Srinivasan ‘04

10. Application: Streaming Lower Bounds • Streaming algorithm: • Example: how many distinct items in the data? • Reduction from Disjointness [Alon, Matias, Szegedy ’99] How much space is required to approximate f(data)? algorithm data

11. Reduction from Disjointness: • Fix a streaming algorithm for Distinct Elements with space , universe size • Construct a protocol for Disj.with elements: algorithm ⇔ #distinct elements in is State of the algorithm and (#bits = )

12. Application 2: KW Games • Circuit depth lower bounds: • How deep does the circuit need to be?

13. Application 2: KW Games • Karchmer-Wigderson’93,Karchmer-Raz-Wigderson’94: find such that

14. Application 2: KW Games • Claim: if has deterministic CC , then requires circuit depth . • Circuit with depth protocol with length

15. Information-Theoretic Lower Bound on Set Disjointness

16. Some Basic Concepts from Info Theory • Entropy of a random variable: • Important properties: • is deterministic • = expected # bits needed to encode

17. Some Basic Concepts from Info Theory • Conditional entropy: • Important properties: • are independent • Example: • If then , if 1then

18. Some Basic Concepts from Info Theory • Mutual information: • Conditional mutual information: • Important properties: • are independent

19. Some Basic Concepts from Info Theory • Chain rule for mutual information: • More generally,

20. Information Cost of Protocols • Fix an input distribution on • Given a protocol , let also denote the distribution of ’s transcript • Information cost of : • Information cost of a function :

21. Information Cost of Protocols • Important property: • Proof: by induction. Let . • : what we know after r rounds what we knew after r-1 rounds what we learn in round r, given what we already know

22. Information vs. Communication • Want: • Suppose is sent by Alice. • What does Alice learn? • is a function of and so • What does Bob learn?

23. Information vs. Communication • Important property: • Lower bound on information cost ⇒ lower bound on communication complexity • In fact, IC lower bounds are the most powerful technique we know

24. Information Complexity of Disj. • Disjointness: is ? • Strategy: for some “hard distribution” , • Direct sum: • Prove that .

25. Hard Distribution for Disjointness • For each coordinate :

26. Let be a protocol for on • Construct for as follows: • Alice and Bob get inputs • Choose a random coordinate , set • Sample and run • For each ,

27. Let be a protocol for on • Construct for as follows: • Alice and Bob get inputs • Choose a random coordinate , set • Bad idea: publicly sample Suppose in , Alice sends . In , Bob learns one bit in he should learn bit But if is public Bob learns 1 bit about !

28. Let be a protocol for on • Construct for as follows: • Alice and Bob get inputs • Choose a random coordinate , set • Another bad idea: publicly sample , Bob privately samples given • But the players can’t sample , independently…

29. Let be a protocol for on • Construct for as follows: • Alice and Bob get inputs • Choose a random coordinate , set Publicly sample Privately sample Privately sample Publicly sample

30. Direct Sum Theorem • Transcript of • Need to show:

31. Information Complexity of Disj. • Disjointness: is ? • Strategy: for some “hard distribution” , • Direct sum: • Prove that . 

32. Hardness of And transcript on should be “very different”

33. Hellinger Distance • Examples: • If have disjoint support,

34. Hellinger Distance • Hellinger distance is a metric • , with equality iff • Triangle inequality:

35. Hellinger Distance • If for some we have then

36. Hellinger Distance vs. Mutual Info • Let be two distributions • Select by choosing , then drawing • Then

37. Hardness of And Same for Bob until Alice acts differently Same for Alice until Bob acts differently

38. “Cut-n-Paste Lemma” • Recall: • Enough to show: we can write

39. “Cut-n-Paste Lemma” • We can write • Proof: • induces a distribution on “partial transcripts” of each length : probability that first bits are • By induction: • Base case: • Set

40. “Cut-n-Paste Lemma” • Step: • Suppose after it is Alice’s turn to speak • What Alice says depends on: • Her input • Her private randomness • The transcript so far, • So • Set

41. Hardness of And

42. Multi-Player Communication Complexity

43. The Coordinator Model sites • bits …

44. Multi-Party Set Disjointness • Input: • Output: is ? • Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13: lower bound of bits

45. Reduction from Disjtograph connectivity • Given we want to • Choose vertices • Design inputs such that is connected iff

46. Reduction from Disjtograph connectivity (Players) (Elements) input graph connected

47. Other Stuff • Distributed computing

48. Other Stuff • Compressing down to information cost • Number-on-forehead lower bounds • Open questions in communication complexity