Pseudorandom Generators for Halfspaces

1. Pseudorandom Generators for Halfspaces Yi Wu (CMU) Joint work with ParikshitGopalan (MSR SVC) Ryan O’Donnell (CMU) David Zuckerman (UT Austin) TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAA

2. Outline • Introduction • Pseudorandom Generators • Halfspaces • Pseudorandom Generators for Halfspaces • Our Result • Proof • Conclusion

3. Deterministic Algorithm Program Input Output The algorithm deterministically outputs the correct result.

4. Randomized Algorithm Program Input Output Random Bits. The algorithm outputs the correct result with high probability.

5. Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Primality testing ST-connectivity Order statistics Searching Polynomial and matrix identity verification Interactive proof systems Faster algorithms for linear programming Rounding linear program solutions to integer Minimum spanning trees shortest paths minimum cuts Counting and enumeration Matrix permanent Counting combinatorial structures Randomized Algorithms

6. Is Randomness Necessary? • Open Problem: • Can we simulate every randomized polynomial time algorithm by a deterministic polynomial time algorithm (the “BPP P” cojecture)? • Derandomization of randomized algorithms. • Primality testing [AKS] • ST-connectivity [Reingold] • Quadratic residues [?]

7. How to generate randomness? Question: How togenerate randomness for every randomized algorithm? Simpler Question: How to generate “pseudorandomness” for some class of programs?

8. Pseudorandom Generator (PRG) Both program Answer Yes/No with almost the same probability Yes /No Yes/ No Input Program Input Program n “pseudorandom” bit PRG Quality of the PRG: number of seed n random bit Seed k<<n random bit

9. Why study PRGs? Algorithmic Applications • When k = log (n), we can derandomize the algorithm in polynomial time. • Streaming Algorithm. Complexity Theoretic Implications • Lower Bound of Circuit Class. • Learning Theory.

10. PRG for Classes of Program • Space Bounded Program [Nis92] • Constant-depth circuits [Nis91, Baz07, Bra09] • Halfspaces[DGJSV09, MZ09]

11. Outline • Introduction • Pseudorandom Generators • Halfspaces • Pseudorandom Generators for Halfspaces • Our Result • Proof • Conclusion

12. - - - - - - + + + - - + + + - + + - - + + + + Halfspaces • Halfspaces: Boolean functions h:Rn → {-1,1} of the form h(x) = sgn(w1x1+…+wnxn- θ) where w1,…, wn,θ R. • Well-studied in complexity theory • Widely used in Machine Learning: Perceptron, Winnow, boosting, Support Vector Machines, Lasso, Liner Regression.

13. Product Distribution • For halfspace h(x), x is sampled from some product distribution; i.e., each xi is independently sampled from distribution Di . For example, each Dican be • Uniform distribution on {-1,1} • Uniform distribution on [-1,1] • Gaussian Distribution

14. Index • Introduction • Pseudorandom Generators • Halfspaces • Pseudorandom Generators for Halfspaces • Main Result • Proof • Conclusion

15. PRG for halfspaces Both program Answer Yes/No with almost the same probability Yes/No Yes/No h(x) = sign(w1x1+…+wnxn-θ) h(x) = sign(w1x1+…+wnxn-θ) Pseudorandom Variable x1, x2 …xn PRG x1, x2 …xnfrom some product distribution k<<n random bit

16. Geometric Interpretation, PRG for uniform distribution over [-1,1]2

17. Geometric Interpretation, PRG for uniform distribution over [-1,1]2 Total Number of points = poly(dim) Number of points in the halfspace is proportional to area.

18. - - - - - - + + + - - + + + - + + - - + + + + Application to Machine Learning How many testing points is it enough to estimate the accuracy of the N dimensional linear classifier? Good PRG implies we only need deterministically check the accuracy on a set of poly(N) points!

19. Other Theoretical Applications • Discrepancy Set for Convex Polytopes • Circuit Lower bound on functions of halfspaces • Counting the Solution of Knapsacks

20. Outline • Introduction • Pseudorandom Generator • Halfspace • Pseudorandom Generators for Halfspaces • Our Results • Proof • Conclusion

21. Previous Result [DiGoJaSeVi,MeZu] PRG For Halfspace over uniform distribution on boolean cube ({-1,1}n) with seed length O(log n).

22. Our Results:Arbitrary Product Distributions • PRG for halfspaces under arbitrary product distribution over Rnwith the same seed length. Only requirement: E[xi4] is a constant. • Gaussian Distribution • Uniform distribution on the solid cube. • Uniform distribution on the hypercube. • Biased distribution on the hypercube. • Almost any “natural distribution”

23. Our Results Functions of k-Halfspaces • PRG for the intersections of k-halfspaces with seed length k log (n). • PRG for arbitrary functions of k-halfspaces with seed length k2 log (n).

24. Outline • Introduction • Pseudorandom Generator • Halfspace • Pseudorandom Generators for Halfspaces • Our Result • Proof • Conclusion

25. Key Observation: Dichotomy of Halfspaces • Under product distributions , every halfspace is close to one of the following: • “Dictator” (halfspaces depending on very few variables, e.g. f(x) = sgn(x1)) • “majority”(no variables has too much weight, e.g. f(x) = sgn(x1+x2+x3+…+xn).

26. Dichotomy of weight distribution Weights decreasing fast (Geometrically) Weights are stable after certain index.

27. Weights Decrease fast (Geometrically) Intuition: for sign(2n x1 + 2n-1 x2+ 2n-2 x3 +…xn) If each xi is from {-1,1} , it is just sign(x1).

28. Weights are stable Intuition: for sign(100 x1 + x2 + x3+…xn) Then by for every fixing of x1, it is a majority on the rest of the variables.

29. Our PRG for Halfspace (Rough) • Randomly hashing all the coordinate into groups. • Use 4-wise independent distribution within each group. • If it is “Dictator-like”: All the important variables are in different groups. • If it is “Majority-like” • (x1 + x2 +.. xn ) is close to Gaussian. 4-wise independent Distribution (somehow) can handle Gaussian.

30. Outline • Introduction • Pseudorandom Generator • Halfspace • Pseudorandom Generators for Halfspaces • Our Result • Proof • Conclusion

31. Conclusion • We construct PRG for halfspaces under arbitrary product distribution and functions of k halfspaces with small seed length. Future Work • Building PRG for larger classes of program; e.g., Polynomial Threshold function (SVM with polynomial kernel).