The Smoothed Analysis of Algorithms: Simplex Methods and Beyond

1. The Smoothed Analysis of Algorithms:Simplex Methods and Beyond Shang-Hua Teng Boston University/Akamai Joint work with Daniel Spielman (MIT)

2. Outline Why What Simplex Method Numerical Analysis Condition Numbers/Gaussian Elimination Conjectures and Open Problems

3. Motivation for Smoothed Analysis Wonderful algorithms and heuristics that work well in practice, but whose performance cannot be understood through traditional analyses. worst-case analysis: if good, is wonderful. But, often exponential for these heuristics examines most contrived inputs average-case analysis: a very special class of inputs may be good, but is it meaningful?

4. Random is not typical

5. Analyses of Algorithms: worst case maxx T(x) average case Er T(r) smoothed complexity

6. Instance of smoothed framework x is Real n-vector sr is Gaussian random vector, variance s2 measure smoothed complexity as function of n and s

7. Complexity Landscape run time input space

8. Complexity Landscape worst case run time input space

9. average case Complexity Landscape worst case run time input space

10. Smoothed Complexity Landscape run time input space

11. Smoothed Complexity Landscape run time smoothed complexity input space

12. Smoothed Analysis of Algorithms • Interpolate between Worst case and Average Case. • Consider neighborhood of every input instance • If low, have to be unlucky to find bad input instance

13. Motivating Example: Simplex Method for Linear Programming max zT x s.t. A x £ y • Worst-Case: exponential • Average-Case: polynomial • Widely used in practice

14. Carbs Protein Fat Iron Cost 1 slice bread 30 5 1.5 10 30¢ 1 cup yogurt 10 9 2.5 0 80¢ 2tsp Peanut Butter 6 8 18 6 20¢ US RDA Minimum 300 50 70 100 The Diet Problem Minimize 30 x1 + 80 x2 + 20 x3 s.t. 30x1 + 10 x2 + 6 x3  300 5x1 + 9x2 + 8x3  50 1.5x1 + 2.5 x2 + 18 x3  70 10x1 + 6 x3  100 x1, x2, x3  0

15. The Simplex Method opt start

16. History of Linear Programming • Simplex Method (Dantzig, ‘47) • Exponential Worst-Case (Klee-Minty ‘72) • Avg-Case Analysis (Borgwardt ‘77, Smale ‘82, Haimovich, Adler, Megiddo, Shamir, Karp, Todd) • Ellipsoid Method (Khaciyan, ‘79) • Interior-Point Method (Karmarkar, ‘84) • Randomized Simplex Method (mO(d) ) (Kalai ‘92, Matousek-Sharir-Welzl ‘92)

17. max zT x s.t. A x £ y max zT x s.t. G is Gaussian Smoothed Analysis of Simplex Method [Spielman-Teng 01] Theorem: For all A, simplex method takes expected time polynomial

18. Shadow Vertices

19. Another shadow

20. Shadow vertex pivot rule start z objective

21. Theorem:For every plane, the expected size of the shadow of the perturbed tope is poly(m,d,1/s )

22. Polar Linear Program z max  zÎ ConvexHull(a1, a2, ..., am)

23. Initial Simplex Opt Simplex

24. Shadow vertex pivot rule

26. Count facets by discretizingto N directions, N

27. [ ] Different Facets < c/N Pr Count pairs in different facets So, expect c Facets

28. Expect cone of large angle

29. Intuition for Smoothed Analysis of Simplex Method After perturbation, “most” corners have angle bounded away from flat opt start most: some appropriate measure angle: measure by condition number of defining matrix

30. Condition number at corner Corner is given by Condition number is • sensitivity of x to change in C and b • distance of C to singular

31. Condition number at corner Corner is given by Condition number is

32. Connection to Numerical Analysis Measure performance of algorithms in terms of condition number of input Average-case framework of Smale: 1. Bound the running time of an algorithm solving a problem in terms of its condition number. 2. Prove it is unlikely that a random problem instance has large condition number.

33. Connection to Numerical Analysis Measure performance of algorithms in terms of condition number of input Smoothed Suggestion: 1. Bound the running time of an algorithm solving a problem in terms of its condition number. 2’. Prove it is unlikely that a perturbed problem instance has large condition number.

34. Edelman ‘88: for standard Gaussian random matrix Condition Number Theorem: for Gaussian random matrix variance centered anywhere [Sankar-Spielman-Teng 02]

35. Edelman ‘88: for standard Gaussian random matrix Condition Number Theorem: for Gaussian random matrix variance centered anywhere (conjecture) [Sankar-Spielman-Teng 02]

36. Gaussian Elimination • A = LU • Growth factor: • With partial pivoting, can be 2n • Precision needed (n )bits • For every A,

37. Condition Number and Iterative LP Solvers Renegar defined condition number for LP maximize subject to • distance of (A, b, c) to ill-posed linear program • related to sensitivity of x to change in (A,b,c) Number of iterations of many LP solvers bounded by function of condition number: Ellipsoid, Perceptron, Interior Point, von Neumann

38. Smoothed Analysis of Perceptron Algorithm [Blum-Dunagan 01] Theorem: For perceptron algorithm Bound through “wiggle room”, a condition number Note: slightly weaker than a bound on expectation

39. Compare: worst-case complexity of IPM is iterations, note Smoothed Analysis of Renegar’s Cond Number Theorem: [Dunagan-Spielman-Teng 02] Corollary: smoothed complexity of interior point method is for accuracy e

40. Perturbations of Structured and Sparse Problems Structured perturbations of structured inputs perturb Zero-preserving perturbations of sparse inputs perturb non-zero entries Or, perturb discrete structure…

41. Goals of Smoothed Analysis Relax worst-case analysis Maintain mathematical rigor Provide plausible explanation for practical behavior of algorithms Develop a theory closer to practice http://math.mit.edu/~spielman/SmoothedAnalysis

42. Geometry of

43. Geometry of should be d1/2 (union bound)

44. Improving bound on For , Apply to random conjecture Lemma: So

45. Compare: worst-case complexity of IPM is iterations, note Smoothed Analysis of Renegar’s Cond Number Theorem: [Dunagan-Spielman-Teng 02] Corollary: smoothed complexity of interior point method is for accuracy e conjecture