1 / 20

Rounding Sum of Squares Relaxations

Joint work with Jonathan Kelner (MIT) and David Steurer (Cornell). Rounding Sum of Squares Relaxations. Boaz Barak – Microsoft Research. workshop on semidefinite programming and graph algorithms. February 10-14, 2014. This talk is about.

gelsey
Download Presentation

Rounding Sum of Squares Relaxations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Joint work with Jonathan Kelner (MIT) and David Steurer (Cornell) Rounding Sum of Squares Relaxations Boaz Barak – Microsoft Research workshop on semidefinite programming and graph algorithms February 10-14, 2014

  2. This talk is about • Semi-definite programming , SOS/Positivstellensatz method • Proof complexity • The Unique Games Conjecture • Graph partitioning, small set expansion • Machine Learning • Cryptography.. (in spirit).

  3. Sum-of-Squares (SOS) Algorithm [Shor’87,Parillo ’00, Nesterov ’00, Lasserre ’01] Motivation: Sometimes a polynomial can have exponentially many local minima… E.g. … but there is still a short proof that E.g. … and this proof can be found efficiently via semidefinite programming (SDP)

  4. SOS Algorithm: • For low degree we consider the program : SOS Proof that : Polynomials and SOS polys s.t. Positivstellensatz: All true bounds have SOS proof. [Artin ’27, Krivine ’64, Stengle ‘74] Degree of proof: max degree of [Gregoriev-Vorobjov’99] Theorem: [Shor ’87, Parillo ’00, Nesterov ’00, Lasserre’01] 1) A proof of degree can be found in time. 2) Can find in time the min s.t. degree d proof that ( Typical CS domains: degree . Examples where degree [Gregoriev’01] )

  5. SOS Algorithm: Program : • For low degree we consider the program : Can optimize in time over programs with degree proofs. SOS Proof that : Polynomials and SOS polys s.t. Positivstellensatz: All true bounds have SOS proof. [Artin ’27, Krivine ’64, Stengle ‘74] Degree of proof: max degree of [Gregoriev-Vorobjov’99] Theorem: [Shor ’87, Parillo ’00, Nesterov ’00, Lasserre’01] 1) A proof of degree can be found in time. 2) Can find in time the min s.t. degree d proof that ( Typical CS domains: degree . Examples where degree [Gregoriev’01] )

  6. Program : SOS Proof that : Can optimize in time over programs with degree proofs. Can’t hope for always: Captures SAT, CLIQUE, 3COL, MAX-CUT, etc… This talk: General method to analyze the SOS algorithm. [B-Kelner-Steurer’13] But maybe often? Essentially only one (robust) lower bound showing [Grigoriev’01] Applications: Optimizing polynomials w/ non-negative coefficients over sphere. Algorithms for quantum separability problem[Brandao-Harrow’13] Sparse coding: learning dictionaries beyond the barrier. Finding sparse vectors in subspaces. Approach to refute the Unique Games Conjecture.

  7. Program : SOS Proof that : Can optimize in time over programs with degree proofs. Can’t hope for always: Captures SAT, CLIQUE, 3COL, MAX-CUT, etc… This talk: General method to analyze the SOS algorithm. [B-Kelner-Steurer’13] But maybe often? Essentially only one (robust) lower bound showing [Grigoriev’01] Applications: Optimizing polynomials w/ non-negative coefficients over sphere. Algorithms for quantum separability problem[Brandao-Harrow’13] Sparse coding: learning dictionaries beyond the barrier. Finding sparse vectors in subspaces. Approach to refute the Unique Games Conjecture.

  8. Program : Finding is hard. We consider easier problem: Given many ’s maximizing , find a single with value close to maximum. (multi) set of s.t., “Finding a needle in a needle-stack” Non-trivial combiner: Only depends on low degree marginals of Combiner Single s.t. , [B-Kelner-Steurer’13]: Transform “simple” non-trivial combiners to algorithm for original problem. Idea in a nutshell: Simple combiners will output a solution even when fed “fake marginals”.

  9. Pseudoexpectations (aka “Fake Marginals”) Def: [Lasserre ’01] Degree pseudoexpectationis operator mapping any degree poly into a number satisfying: • Program : Finding is hard. We consider easier problem: Normalization: Given many ’s maximizing , find a single with value close to maximum. Linearity: of deg Positivity: of deg (multi) set of s.t., “Finding a needle in a needle-stack” Non-trivial combiner: Only depends on low degree marginals of Fundamental Fact: deg SOS proof for for any degpseudoexpectation operator Combiner Take home message: Pseudoexpectation “looks like” real expectation to low degree polynomials. Can efficiently find pseudoexpectation matching any polynomial constraints. Proofs about real random vars can often be “lifted” to pseudoexpectation. Single s.t. , [B-Kelner-Steurer’13]: Transform “simple” non-trivial combiners to algorithm for original problem. Idea in a nutshell: Simple combiners will output a solution even when fed “fake marginals”. “fake marginals”.

  10. Combining Rounding Pseudoexpectations (aka “Fake Marginals”) Problem: Given low degree maximize s.t. Def: [Lasserre ’01] Degree pseudoexpectationis operator mapping any degree poly into a number satisfying: • Program : Finding is hard. We consider easier problem: Normalization: Given many ’s maximizing , find a single with value close to maximum. Linearity: of deg Positivity: of deg (multi) set of s.t., “Finding a needle in a needle-stack” Non-trivial combiner: Only depends on low degree marginals of Fundamental Fact: deg SOS proof for for any degpseudoexpectation operator Combiner Take home message: Pseudoexpectation “looks like” real expectation to low degree polynomials. Can efficiently find pseudoexpectation matching any polynomial constraints. Proofs about real random vars can often be “lifted” to pseudoexpectation. Single s.t. , [B-Kelner-Steurer’13]: Transform “simple” non-trivial combiners to algorithm for original problem. Idea in a nutshell: Simple combiners will output a solution even when fed “fake marginals”. “fake marginals”.

  11. Combining Rounding Problem: Given low degree maximize s.t. Non-trivial combiner: Algwith Input: , r.v. over s.t. Output: s.t. Crucial Observation: If proof that is good solution is in SOS framework, then it holds even if is fed with a pseudoexpectation. Corollary: In this case, we can find efficiently: Use SOS PSD to find pseudoexpectation matching input conditions. Use to roundthe PSD solution into an actual solution [B-Kelner-Steurer’13]: Transform “simple” non-trivial combiners to algorithm for original problem.

  12. Example Application: Dictionary Learning / Sparse Coding Let set of vectors. Goal: Given examples of form , where recover Find the “right” representation of observed data [Olhausen-Field ’96] LOTS of work: important primitive in Machine Learning, Vision, Neuroscience... Previous best (rigorous) results:[Spielman-Wang-Wright ’12, Arora-Moitra-Ge ‘13, Agrawal-Anandkumar-Jain-Netrapalli-Tandon ‘13] We show:is sufficient* (even in non-independent, overcomplete case) * In quasipoly time; we show is sufficient in poly time.

  13. Example Application: Dictionary Learning / Sparse Coding Let set of vectors. Goal: Given examples of form , where recover Find the “right” representation of observed data [Olhausen-Field ’96] LOTS of work: important primitive in Machine Learning, Vision, Neuroscience,… Previous best (rigorous) results:[Spielman-Wang-Wright ’12, Arora-Moitra-Ge ‘13, Agrawal-Anandkumar-Jain-Netrapalli-Tandon ‘13] We show:is sufficient* (even in non-independent, overcomplete case) * In quasipoly time; we show is sufficient in poly time.

  14. Let set of vectors. Goal: Given examples of form , where recover Achieve in 3 steps: (1) Find a program s.t. every maximizing is close to one of ’s (2) Give combining alg taking moments of dist over maximizers into a vector close to one of ’s. Result generalizes to overcomplete, non independent case. (3) Show that arguments in (1) and (2) fall under the SOS framework. Step 1. For simplicity, assume , ’s orthonormal basis, i.i.d. random vars over s.t. (can approximate arbitrarily well from examples) Consider the polynomial Opening parenthesis we get Corollary:unit, Establishes (1) !

  15. Let set of vectors. Goal: Given examples of form , where recover Achieve in 3 steps: (1) Find a program s.t. every maximizing is close to one of ’s (2) Give combining alg taking moments of dist over maximizers into a vector close to one of ’s. Result generalizes to overcomplete, non independent case. (3) Show that arguments in (1) and (2) fall under the SOS framework. Step 1. For simplicity, assume , ’s orthonormal basis, i.i.d. random vars over s.t. (can approximate arbitrarily well from examples) Consider the polynomial Opening parenthesis we get Corollary:unit, Establishes (1) !

  16. Let set of vectors. Goal: Given examples of form , where recover Achieve in 3 steps: (1) Find a program s.t. every maximizing is close to one of ’s (2) Give combining alg taking moments of dist over maximizers into a vector close to one of ’s. Slightly tedious but straightforward computations. Happens w prob (3) Show that arguments in (1) and (2) fall under the SOS framework. Step 2. Let be dist over unit vectors s.t. every satisfies for some Pick set of random (stdgaussian) vectors. Let be matrix s.t. for (Note that ) Our combining algorithm outputs the top e-vec of . Suppose that and for every , . Then if then (up to scaling) and we’ll succeed. Establishes (2) !

  17. A personal overview of the Unique Games Conjecture Unique Games Conjecture: UG/SSE problem is NP-hard. [Khot’02,Raghavendra-Steurer’08] reasons to believe reasons to suspect “Standard crypto heuristic”: Tried to solve it and couldn’t. Random instances are easy via simple algorithm[Arora-Khot-Kolla-Steurer-Tulsiani-Vishnoi’05] Very clean picture of complexity landscape:simple algorithms are optimal[Khot’02…Raghavendra’08….] SOS proof system Quasipolyalgo on KV instance[Kolla ‘10] Simple poly algorithms can’t refute it[Khot-Vishnoi’04] Subexponential algorithm [Arora-B-Steurer ‘10] Simple subexp' algorithms can’t refute it[B-Gopalan-Håstad-Meka-Raghavendra-Steurer’12] SOS solves all candidate hard instances[B-Brandao-Harrow-Kelner-Steurer-Zhou ‘12] SOS useful for sparse vector problemCandidate algorithm for search problem[B-Kelner-Steurer ‘13]

  18. Conclusions • Sum of Squares is a powerful algorithmic framework that can yield strong results for the right problems.(contrast with previous results on SDP/LP hierarchies, showing lower bounds when using either wrong hierarchy or wrong problem.) • “Combiner” view allows to focus on the features of the problem rather than details of relaxation. • SOS seems particularly useful for problems with some geometric structure, includes several problems related to unique games and machine learning. • Still have only rudimentary understanding when SOS works or not. • Other proof complexity approximation algorithms connections?

  19. Other Results Sparse vector problem: Recover -sparse vector in -dimensional subspace given arbitrary basis. (motivation: machine learning, optimization , [Demanet-Hand 13]worst-case variant is algorithmic bottleneck in UG/SSE alg[Arora-B-Steurer’10]) Random case: Recovery for any (Improving on [Demanet-Hand ‘13]) Worst case: Recovery* for [Brandao-Harrow’12]: Using our techniques, find separable quantum state maximizing a “local operations classical communication” () measurement.

More Related