A Combinatorial, Primal-Dual Approach to Semidefinite Programs

1. A Combinatorial, Primal-Dual Approach to Semidefinite Programs Satyen Kale Microsoft Research Joint work with Sanjeev Arora Princeton University

2. Max Cut [GW’95] Balanced Partitioning [ARV’04] (a Ç:b) Æ (:a Ç c) Æ (a Ç b) Æ (:c Ç b) 1 0 0 0 1 1 0 1 Graph Coloring [KMS’98, ACC’06] Constraint Satisfaction [ACMM’05] The Ubiquity of Semidefinite Programming dx(t)/dt = Ax(t) Control Theory SDP

3. MISSING Combinatorial, Primal-Dual algorithms A1² X ¸ b1  Am² X ¸ bm i wi(Ai² X – bi) ¸ 0 Algorithms for SDP • Interior point methods [NN’90, A’95]: • O(√m(n3 + m)) time • Ellipsoid method [GLS’81]: • O(n8) iterations • Lagrangian Relaxation [KL’95], [AHK’05]: • Reduction to eigenvectors • poly(1/) dependence on, limits • applicability (e.g. Sparsest Cut)

4. Primal-Dual algorithms for LP:Multicommodity Flows Unit capacities Objective:Maximize total flow, while respecting edge capacities

5. 1 1 1 1 1 1 1 1 1 1 1 1 1 Primal-Dual algorithms for LP:Multicommodity Flows 1 +  1 • Edge weights we = 1 • Repeat until some e reaches capacity: • Find shortest path p • Route 2/log(m) flow on p • Update we for all e 2 p as weà we¢ (1 + ) • Output flow. Thm [GK’98]: Stops in Õ(m) rounds with (1 – 2)¢OPT flow. Total running time is Õ(m2).

6. Primal-Dual algorithms for LP:Multicommodity Flows • Primal-Dual algorithm • Combinatorial • Multiplicative Weights Update Rule • Edge weights we = 1 • Repeat until some e reaches capacity: • Find shortest path p • Route 2/log(m) flow on p • Update we for all e 2 p as weà we¢ (1 + ) • Output flow. Thm [GK’98]: Stops in Õ(m) rounds with (1 – 2)¢OPT flow. Total running time is Õ(m2).

7. Starting point for our work: Analogous primal-dual algorithms for SDP? • Difficulties: • Positive semidefiniteness hard to maintain • Rounding algorithms exploit geometric structure (e.g. negative-type metrics) • Matrix operations inefficient to implement • Our work: a generic scheme that yields fast, combinatorial, primal-dual algorithms for various optimization problems using SDP

8. Our Results: Primal-Dual algorithms [AHK’04] [AHK’04]

9. Our Results: Primal-Dual algorithms [AHK’04] Cannot be achieved by LPs even! [AHK’04]

10. Our Results: Near-linear time algorithm for Max Cut • Õ(n + m) time algorithm to approximate Max Cut SDP to (1 + o(1)) factor • Previous best: Õ(n2) time algorithm by Klein, Lu ‘95

11. Prototype MW for LPs: Winnow [L’88] - -  - a1¢ x ¸ a2¢ x ¸  am¢ x ¸ x ¸ 0 i xi = 1 0 0  0 Thm: Finds x in O(2log(n)/2) iterations. = maxij |aij| MW for LP: Init: x = (1/n, …, 1/n) Update: xj = xj¢ exp(- aij)/  = norm. factor x Oracle Find i s.t. ai¢ x < - ai Convex comb. of constraints allowed: i yi ai¢ x < -, where yi¸ 0, i yi = 1. Hence Primal-Dual. MW algorithm: boosting, hard-core sets, zero-sum games, flows, portfolio, … Analysis uses  as potential fn.

12. Prototype Matrix MW for SDPs - -  - A1² X ¸ A2² X ¸  Am² X ¸ X º 0 Tr(X) = 1 0 0  0 Thm: Finds X in O(2log(n)/2) iterations. = maxikAik Matrix MW for SDP: Init: X = I/n Update: X= exp(-ts=1 Ais)/  = norm. factor X Oracle Find it s.t. Ait² X < - Ait Convex comb. of constraints allowed: i yi Ai²X < -, where yi¸ 0, i yi = 1. Hence Primal-Dual. Analysis uses  as potential fn.

13. The Matrix Exponential • Matrix exponential: exp(A) = I + A + A2/2! + A3/3! + … • Always PSD:exp(A) º 0 • Tricky beast: exp(A+B) = exp(A) exp(B) • Computation: • O(n3) time • Usually, can use J-L lemma • Only need exp(A)v products: Õ(m) time Golden-Thompson inequality: Tr(exp(A+B)) · Tr(exp(A)exp(B)) Matrix MW for SDP: Init: X = I/n Update: X= exp(-ts=1 Ais)/  = norm. factor

14. Input v2 v1 SDP Opt vn Approximation via SDP Relaxations 0-1 Quadratic Program Reduction Relaxation Rounding Algorithm Primal-Dual SDP SDP: Vector vars SDP Solver

15. Input v2 v1 vn Primal-Dual SDP Framework 0-1 Quadratic Program Reduction Rounding Algorithm Relaxation Primal-Dual SDP y1,…, ym Matrix MW SDP: Vector vars X

16. Approximating Balanced Separator Cut (S, S’) is c-balanced if |S|, |S’| ¸cn Min c-Balanced Separator:c-balanced cut of min capacity Numerous applications: Divide-and-conquer algorithms Markov chains Geometric embeddings Clustering Layout problems … S S’ G = (V, E)

17. SDP for Balanced Separator ¼kvi – vjk2 = 0 if i, j on same side, = 1 otherwise SDP: min i,j 2 E ¼kvi – vjk2 8i: kvik2 = 1 8ijk: kvi–vjk2 + kvj–vkk2¸kvi–vkk2 i, j ¼kvi – vjk2 ¸ c(1-c)n2 G = (V, E) S S’ vi = -1 vi = 1

18. Implementing Oracle Primal: mini,j 2 E ¼kvi – vjk2 8i: kvik2 = 1 8ijk: kvi–vjk2 + kvj–vkk2¸kvi–vkk2 i, j ¼kvi – vjk2 ¸ c(1-c)n2 Oracle: given X (thus, vi), : check first and third constraints  inequalities: dual vars , multicommodity flow s.t. total flow from any node is · d no capacity is exceeded i,j fijkvi – vjk2¸ Bounds eigenvalues viademand graph Laplacian • Thm: Givenvi, : • Max-flow)desired flow or cut of • value O(log(n)¢). • Multicommodity flow) desired flow • or cut of value O(√log(n)¢). fij = total flow between i, j

19. Conclusions and Future Work • Matrix MW algorithm: more applications in • Solving SDPs: e.g. Min Linear Arrangement • Quantum algorithms: density matrix is a central concept • Learning: e.g. online variance minimization [WK COLT’06], online PCA [WK NIPS’06] • Other applications? • Linear time algorithm for Sparsest Cut? • Our algorithm runs in Õ(m + n1.5) time for an O(log n) approximation

20. Thank you!