Semidefinite Programming and Approximation Algorithms for NP-hard Problems: A Survey

1. Semidefinite Programming and Approximation Algorithms for NP-hard Problems: A Survey Sanjeev AroraPrinceton University http://intractability.princeton.edu Arora: SDP + Approx Survey

2. NP-completeness Thousands of problems are NP-complete (TSP, Scheduling, Circuit layout, Machine Learning,..) Pragmatic Researcher “Why the fuss? I am perfectly content with approximatelyoptimal solutions.” (e.g., cost within 10% of optimum) Good news: Possible for a few problems. (“Approximation Algorithms”) Bad News: NP-hard for many problems. (“PCPs”) Arora: SDP + Approx Survey

3. Approximation Algorithms MAX-3SAT: Given 3-CNF formula , find assignment maximizing the number of satisfied clauses. An -approximation algorithm is one that for every formula, produces in polynomial time an assignment that satisfies at least OPT/ clauses. ( > 1). Good News: [KarloffZwick’97] 8/7-approximation algorithm. Bad News: [Hastad’97] If P  NP then for every  > 0, an(8/7 -)-approximation algorithm does not exist. Many similar results... Arora: SDP + Approx Survey

4. Example: 2-approximation for Min Vertex Cover G= (V, E) Vertex Cover = Set of vertices that touches every edge “LP Relaxation” most Claim: Value at least OPT/2 0 · xi· 1 Proof:On Complete Graph Kn, OPT = n-1 but setting all xi = 1/2 gives feasible LP soln Proof: “Rounding” Arora: SDP + Approx Survey

5. General Philosophy… Interested in: NP-hard Minimization Problem Value = OPT Write tractable relaxationvalue= Round to get a solution of cost = Approximation ratio = Integrality gap Arora: SDP + Approx Survey

6. SDP = Generalization of linear programming; “vector programming” Graph Vector Representation.(Inner products satisfysome linear constraints) Developed in 1970s as one of many flavors of nonlinear optimization. Can be solved in poly time (GLS’81).Has many applications in operations research, control theory, approximation algorithms for NP-hard problems. Arora: SDP + Approx Survey

7. Main Idea in SDP: “Simulate” nonlinear programming by convex program Nonlinear program for Vertex Cover Homogenized SDP relaxation: New variable intended to stand for “Vector Programs.” Arora: SDP + Approx Survey

8. Take home message… • SDP gives best approximation known for host of NP-hard problems (and algorithms can be made highly efficient): • Vertex Cover • Sparsest Cut and most graph partitioning problems • Graph coloring • Max-cut, and every “Constraint Satisfaction Problem”…. Analysis of these algorithms used interesting geometric ideas, which have had other applications. Compelling evidence from complexity theory that no poly-timealgorithm can do better than many of these SDP-based algorithms.(Novel interplay between SDP, reductions, high-dimension geometry….) Arora: SDP + Approx Survey

9. Outline: SDPs & Approximation • SDP and its use in approximation: Generations 1 & 2 • Understanding SDPs <-> high dimensional geometry • Faster algorithms (multiplicative update rule) • Limitations of SDPs, Unique Games Conjecture • Future directions • Open problems Arora: SDP + Approx Survey

10. How do you understand thesevector programs? Ans. Interesting geometric analysis Arora: SDP + Approx Survey

11. Understanding SDPs <--> Understanding phenomena in high-dimensional geometry Vertex Cover SDP computes c-approximation for c < 2 iff following is true [GK96] Vertices: n unit vectorsEdges: almost-antipodal pairs Rn Every graph in this family has an independent set of size Thm [Frankl-Rodl’87] False. Arora: SDP + Approx Survey

12. SDP rounding: The two generations Generation 1: *Uses random hyperplane as in [GW]; * Edge-by-edge analysis Max-2SAT and Max-CUT [GW’94] ;Graph coloring [KMS’95]; MAX-3SAT [KZ’97]; Algorithms for Unique Games;.. Generation 2: Global rounding and analysisGraph partitioning problems [ARV’04], Graph deletion and directed partitioning problems [ACMM’05], New analysis of graph coloring [ACC’06] Disproof of UGC for expanding constraints [AKKSTV’08] Recently, generation 1.5: “Squish ‘n solve” rounding. k-CSPs [RS’09] Arora: SDP + Approx Survey

13. G = (V,E) Find that maximizes capacity . Quadratic Programming Formulation 1st Generation Rounding: Ratio 1.13.. for MAX-CUT [Goemans-Williamson’93] Semidefinite Relaxation [DP ’91, GW ’93] Arora: SDP + Approx Survey

14. vi ij vj Randomized Rounding (1st Gen) [GW ’93] Rn v2 v1 v6 Form a cut by partitioning v1,v2,...,vn around a random hyperplane. v3 v5 SDPOPT Old math rides to the rescue... Arora: SDP + Approx Survey

15. Surely this bizarre algorithm is not the “right” way to solve max cut?? Arora: SDP + Approx Survey

16. Fact 1: No rounding algorithm can produce a better solution out of this SDP [Feige-Schechtman] “Edges between all pairs of vectorsmaking an angle 138 degrees.” Fact 2: If P NP then impossible to get 1.06-approximation in poly time [Hastad’97] Fact 3: If “unique games conjecture” is true, no better than 1.13-approximation possible in poly time.[KKMO’05] (i.e., algorithm on prev. slide is optimal) Arora: SDP + Approx Survey

17. 2nd Generation: for c-balanced separator G= (V, E); constant c >0 1 -1 Goal: Find cut s.t. each side contains at least c fraction of nodes and minimized SDP: “Triangle inequality” Angle subtended by the line joiningtwo of them on the third is non-obtuse; “ “ condition. Arora: SDP + Approx Survey

18. 4. Call remaining sets S, T. Do BFS from S to T according to distance 3. Remove any pair (i, j) that lie on opp.sides of slab but T S 2. Remove points in “slab” of width Rounding algorithm for -approximation [ARV’04] 1. Pick random hyperplane S T 5. Output level of BFS tree with least # of edges. Heart of analysis: Shows |S|, |T| = W(n);“Large well-separated sets” Arora: SDP + Approx Survey

19. If then no graph in this family is an “expander.” Geometric fact underlying the analysis (restatement of [ARV04] “Structure Theorem” by [AL06]) Vertices: unit vectorssatisfying “triangle inequality” = (“expander” : |(S)|(|S|) ) Edges: Proof: Difficult “chaining” argument. (Aside: Has been used to prove that l1 embeds into l2 with distortion[CGR’05,ALN’06]) Arora: SDP + Approx Survey

20. Next few slides: Results showing Approximation is hard assuming Unique-Games-Conjecture (UGC) Recall: Integrality gap of an SDP = min c st <= c Let “tough instance” = problem instance with integrality gap cThese play crucial role in above reductions!! Arora: SDP + Approx Survey

21. Unique Games Given: Number p, and m equations in n vars of the form: Promise: Either there is a solution that satisfies fractionof constraints or no solution satisfies even fraction. UGC [Khot’02]: Deciding which case holds is NP-hard. [Raghavendra’08; building upon [KKMO’04][MOO’05]] UGC  For every MAX-CSP, the simplest SDP relaxation is the best possible poly-time approximation. Arora: SDP + Approx Survey

22. Equations Anatomy of a UGC-based hardness result(eg Khot-Regev, KKMO, Raghavendra08) Variables Interpret as a graph Prove using harmonicanalysis that near--optimum solnscorrespond to good solution to theunique game Equations Variables Replace edges/vertices with gadgets involving “integrality gap instance” Arora: SDP + Approx Survey

23. Generation 1.5 rounding: Squish-n-solve(Raghavendra-Steurer’09; provably optimal approximation for all k-CSPs if UGC is true) Project to random t-dimsubspace; t =O(1) Merge nodeswhose vectorsare close together; get instance ofsize exp(t) andsolve it optimally. If g= approx. ratio for this algorithm then it is also theintegrality gap for SDP, and also the best possible approx. ratio (assuming UGC). Arora: SDP + Approx Survey

24. Issue of Running Time Solving SDPs with m constraints takes time. m =n3 in some of these SDPs! Next slide: Often, can reduce running time: O(n2) or O(n3) [AHK’05], [AK’07]even O(n) for CSPs! [St’10]; O(n1.5 +m) for sparsest cut [S’09] Main idea: “Primal-dual schema.” Solve to approximate optimality; using insights from the rounding algorithms. “Multiplicative Weight-Update Rule for psd matrices” Arora: SDP + Approx Survey

25. Primal-dual approach for SDP relaxations (contd.) [A., Kale’07] At step t: Primal player: PSD matrix Xt; candidate primal Let me run the rounding algorithm on Pt, get a primal integer candidate and point out how pitiful it is. Dual player: “Candidate slack matrix” Mt Primal player: Xt+1 = exp(-t Mt) (Analysis uses formal analogy between real #s and symmetric matrics: [Other ingredients: flow computations, eigenvalues, dimension reduction tricks, etc.; simplified by [KRV07], [KRVV08]] Arora: SDP + Approx Survey

26. Future directions…. Arora: SDP + Approx Survey

27. Direction 1: How to soup up the relaxation • Other forms of convex relaxations instead of SDP? (e.g. geometric programming, convex programming) For a start, re-derive existing approximation ratios(eg 0.878.. For MAX-CUT) using the above. Arora: SDP + Approx Survey

28. Direction 2: Understand “Lifted” SDP relaxations Recall: SDP tries to “simulate” nonlinear programming; Variable for Why not take it to the next level? Variablesfor products of up to k variables. This is the main idea of Lovasz-Schrijver’91, Sherali-Adams, Lasserre etc. Yields better approx for hypergraph I.S. [CS’08] Don’t seem to help for some problems, even for k =n/100: MAX-k-SAT [BGHMP’06], [AAT’05], Vertex Cover[ABLT’06],[STT’07a+b] MAX-CUT, Vertex Cover etc. [CMM’08] MAX-3SAT (Sch’09) (v. subtle and beautiful ideas!) Arora: SDP + Approx Survey

29. Direction 3: graph expansion [ARV]; SDP w/ triangle ineq. Approx Ratio O(1) approx is UGC hard; as is improving Cheeger for constt e [Cheeger’72]; eigenvalue Expansion Lots of space for improvement even if UGC holds Also, “small set expansion” problem is v. close to UG and progress on it would possibly give progress on UG. [RS’10] Arora: SDP + Approx Survey

30. Direction 4: Subexponential algorithms • Inspiration: Unique Games with completeness 1-e can be approximately solved in exp(ne) time [ABS’10] (For problems like MAX3SAT, no such algorithms exist if 3SAT has no subexp. algorithms) Intriguing possibility: Many of the UGC-hard problemshave subexponential algorithms. Another interesting idea: derive [ABS’10] algorithm for UGusing SDPs or Lasserre relaxations. Arora: SDP + Approx Survey

31. Direction 5: SDP in Avg. Case Complexity Problems like 3SAT seem difficult not only in the worstcase but also “on average.” (Needs careful definition!) Theory of Avg Case complexity [Levin’84] doesn’t usuallyapply to problems of practical interest (e.g., random 3SAT). Recent development: Interreducibility among some“average case” problems of interest. [Feige’01]; e.g Easy “optimal” 7/8-hardness of MAX-3SAT. SDP is used in the reduction! (Used to weed out “uninteresting” cases) Arora: SDP + Approx Survey

32. Direction 6: Applications to quantum computing • PSD matrix of trace 1 = “density matrix”, a way to describe a mixed quantum state Recent result QIP=PSPACE (JUW’10) uses the [AK’07] Primal-dual framework. (“Fast NC computation of near-optimum quantum state”) Expertise in designing specialized matrices for SDPintegrality gaps may prove useful in QC… Arora: SDP + Approx Survey

33. Open problems • Can Lasserre relaxations compute nontrivial approximations to Vertex Cover, MAX-CUT, etc?(ruled out already for MAX-3SAT [Sch’08]) • Generation 3 rounding? • Iterative rounding for SDPs? • Resolve UGC (eg disprove by giving truly subexp. algorithms) • SDP as a proof technique---apply to open problems of circuit complexity, communication complexity etc. Looking forward to many developments THANK YOU! Arora: SDP + Approx Survey

34. Classical MW update rule (Example: predicting the market) • N “experts” on TV • Can we perform as good as the best expert in hindsight? 1$ won for correct prediction 1$ lost for incorrect prediction Thm[Going back to Hannan, 1950s] Yes. Arora: SDP + Approx Survey

35. Weighted Majority Algorithm (LW’94) LossesM1t M2t M3t … • For each expert, weight wi. Initially wi1 • Follow expert i w/ prob. proportional to wit • Update weights according to Claim: Expected per-round loss of our algorithm Arora: SDP + Approx Survey

36. Lagrangian method to approximately solve LPs (PST’91, many others) LossesM1t M2t M3t … Experts = Dual LP constraints ; maintain weightingLoss vector = Slack vector of candidate dual soln • For each expert, weight wi. Initially wi1 • Follow expert i w/ prob. proportional to wit • Update weights according to Claim: After a few rounds; the average of all the lossvectors is an approximately feasible dual soln. Claim: Expected per round loss of our algorithm Arora: SDP + Approx Survey

37. Lagrangian method to approximately solve LPs (PST’91, many others since) x = weighting of n experts; updated via multiplicative update Loss vector Expected loss ! Only 1 constraint ! Claim: Expected per round loss of our algorithm Average per-round loss of expert i = i’th coordinate of Arora: SDP + Approx Survey

38. Lagrangian method to approximately solve SDPs (A.,Kale ‘07) psd x = weighting of n experts; updated via multiplicative update x= PSD matrix; updated according to Loss vector Expected loss ! Claim: Expected per round loss of our algorithm Arora: SDP + Approx Survey

39. SDPs and MW Updates: Primal-dual algorithm Known: MW Update rule --> Approx. solutions to LPs[PST’91, Y’95, GK’97,..etc.] “experts” <-> constraints“payoffs” <-> “slack in constraint” [AK’07] Matrix MW update rule that uses formal analogybetween psd matrices and nonnegative real #s. [Golden-Thompson] (Spl. Case: LPs= SDPs with 0’s on offdiagonals) [Other ingredients: flow computations, eigenvalues, dimension reduction tricks, etc.] Arora: SDP + Approx Survey

40. Embeddings and Cuts Thm[LLR94, AR94]: Integrality gap for SDP for Nonuniform Sparsest Cut = Min distortion of any embedding of into Rounding algorithm of [ARV04] gives insight into structureof ; basis of new embeddings Hardness results for sparsest cut yielded insights at the heart of the embedding impossibility results. Arora: SDP + Approx Survey

41. Limitations of SDPs For many problems, we know neither an NP-hardness result (via PCPs) nor a good SDP-based approach. Can we show that known SDPs don’t work?? 1st generation results: Specific SDPs don’t work 2nd Generation results: Large families of LPs or SDPs don’t work [ABL’02], [ABLT’06]: “Proving integrality gaps without knowing the LP.” Much subsequent work, especially on families obtained from “lift and project” ideas) Arora: SDP + Approx Survey

42. Example: 2-approximation for Min Vertex Cover G= (V, E) Vertex Cover = Set of vertices that touches every edge “LP Relaxation” most Claim: Value at least OPT/2 Proof:On Complete Graph Kn, OPT = n-1 but setting all xi = 1/2 gives feasible LP soln Proof: “Rounding” Arora: SDP + Approx Survey

43. Next, briefly Connection between analysis of SDPs and Geometric Embedding of Metric Spaces Arora: SDP + Approx Survey

44. f Geometric embeddings of metric spaces (X, d): metric space x y f(y) d(x, y) f(x) C = distortion Thm (Bourgain’85) For every X, there is f s.t. C= O(log n). Open qs since then: is it possible to achieve smaller C forconcrete X, say X = ? [CGR’05,ALN05]: Yes, C possible for X = Via [LLR94,AR94] implies approx for general sparsest cut Arora: SDP + Approx Survey

45. Main issue: Local versus Global Example: [Erdos] There are graphs on n vertices that cannot be colored with 100 colors yet every subgraph on 0.01 n vertices is 3-colorable. LP relaxations or SDP relaxations concern local conditions. How well do such local conditions capture global property in question? Results for MAX-k-SAT [], [AAT’05], Vertex Cover[ABLT’06], [STT’07a+b] MAX-CUT, Vertex Cover etc. [CMM’08] “Lifted SDPs.” Connections to Proof Complexity. Arora: SDP + Approx Survey