Instructor: Shengyu Zhang

1. CSC3160: Design and Analysis of Algorithms Week 6: Linear Programming Instructor: Shengyu Zhang

2. LP • Motivating examples • Introduction to algorithms • Simplex algorithm • On a particular example • General algorithm • Duality • An application to game theory

3. Example 1: profit maximization • A company has two types of products: P, Q. • Profit: P --- $1 each; Q --- $6 each. • Constraints: • Daily productivity (including both P and Q) is 400 • Daily demand for P is 200 • Daily demand for Q is 300 • Question: How many P and Q should we produce to maximize the profit? • units of P, units of Q

4. How to solve? • units of P units of Q • Constraints: • Daily productivity (including both P and Q) is 400 • Daily demand for P is 200 • Daily demand for Q is 300 • Question: how much P and Q to produce to maximize the profit? • Variables: • and . • Constraints: • Objective:

5. Illustrative figures

6. Example 2 • We are managing a network with bandwidth as shown by numbers on edges. • Bandwidth: max units of flows • 3 connections: AB, BC, CA • We get $3, $2, $4 for providing them respectively. • Two routes for each connection: short and long. • Question: How to route the connections to maximize our revenue?

7. Example 2 : amount of flow of the short route : amount of flow of the long route • Variables: • . • Constraints: • (edge ) • (edge ) • (edge ) • (edge ) • (edge ) • (edge ) • Objective:

8. LP in general • Max/min a linear function of variables • Called the objective function • All constraints are linear (in)equalities • Equational form: max max s.t. s.t. • : variables. • : coefficients in constraints Superscript T: transpose of vectors. Inequality: entry-wise

9. Transformations between forms • Minvs. max: • Inequality directions: • Equalities to inequalities: (ai: row i in matrix A) • , and .

10. Transformations between forms • Inequalitiesto equalities: • The newly introduced variable is called slack variable • “Unrestricted” to “nonnegative constraint”: • unrestricted

11. feasibility • The constraints of the form is a line on the plane of . • ? half space. • All constraints are satisfied: the intersection of these half spaces. --- feasible region. • Feasible region nonempty: LP is feasible • Feasible region empty: LP is infeasible

12. Adding the objective function into the picture • The objective function is also linear • also a line for a fixed value. • Thus the optimization is: try to move the line towards the desirable direction s.t. the line still intersectswith the feasible region.

13. Possibilities of solution • Infeasible: no solution satisfying and . • Example? Picture? • Feasible but unbounded: can be arbitrarily large. • Example? Picture? • Feasible and bounded: there is an optimal solution. • Example? Picture?

14. Three Algorithms for LP • Simplex algorithm (Dantzig, 1947) • Exponential in worst case • Widely used due to the practical efficiency • Ellipsoid algorithm (Khachiyan, 1979) • First polynomial-time algorithm: • : number of input bits • Little practical impact. • Interior point algorithm (Karmarkar, 1984) • More efficient in theory: • More efficient in practice (compared to Ellipsoid). Weakly polynomial time

15. Simplex method: geometric view • Start from any vertex of the feasible region. • Repeatedly look for a better neighbor and move to it. • Better: for the objective function • Finally we reach a point with no better neighbor • In other words, it’s locally optimal. • For LP: locally optimal globally optimal. • Reason: the feasible region is a convex set.

16. A sequence of (simplex) tableaus Pick an initial tableau Update the tableau Terminate What’s a tableau? How? What’s the rule? When to terminate?Why optimal? Simplex algorithm: Framework Complexity?

17. An introductory example • Consider the following LP • The equalities are , , • Let . • Rewrite equalities as follows. (A tableau.)

18. An introductory example • The equalities are , , • Let . • is a basis:is non-singular. • : columns of . • The basis is feasible: . • Rewrite equalities as follows. • Set , and get . • And .

19. An introductory example • Now we want to improve . • Clearly one needs to increase or . • Let’s say . • we keep . • How much can we increase ? • We need to maintain the first three equalities. • Rewrite equalities as follows. • Set , and get . • And .

20. An introductory example • Setting , the first three equalities become • To maintain all , we need and . • obtained from the first and third equalities above. • So can increase to 1. • And becomes 0. • Rewrite equalities as follows. • Set , and update other variables . • And .

21. An introductory example • Now basis becomes • the basis is feasible. • Compare to previous basis , one index (3) leaves and another (2) enters. • This process is called a pivot step. • Rewrite the tableau by putting variables in basis to the left hand side. • Rewrite equalities as follows.

22. An introductory example • Now basis becomes • the basis is feasible. • Compare to previous basis , one index (3) leaves and another (2) enters. • This process is called a pivot step. • Rewrite the tableau by putting variables in basis to the left hand side. • Rewrite equalities as follows.

23. An introductory example • Repeat the process. • To increase , we can increase . • Increasing decreases since the coefficient is negative. • We keep , and see how much we can increase . • We can increase to 1, at which point becomes . • Rewrite equalities as follows. • Set , and update other variables . • And .

24. An introductory example • Rewrite equalities as follows. • Set , and update other variables . • And . • The new basis is . • Rewrite the tableau.

25. An introductory example • The new basis is . • Rewrite the tableau. • See which variable should increase to make larger. • in this case. • See how much we can increase . • . • Update ’s and . • Rewrite equalities as follows. • Set , and update other variables . • And .

26. An introductory example • The new basis is . • Rewrite the tableau. • See which variable should increase to make larger. • None! • Both coefficients for and are negative now. • Claim: We’ve found the optimal solution and optimal value! ☺ • Rewrite equalities as follows. • Set , and update other variables . • And .

27. Formal treatment • Now we make the intuitions formal. • We will rigorously define things like basis, feasible basis, tableau, … • discuss the pivot steps, • and formalize the above procedure for general LP.

28. Basis • In the matrix , a subset is a basis if those columns of in are linearly independent. • In other words, is nonsingular. • Denote . • A basis is feasible if . • The inequality is entry-wise.

29. (Simplex) tableau • A (simplex) tableau w.r.t. feasible basis is the following system of equations

30. Tableau • [Prop 1] If is nonsingular, then satisfies • Proof. • .

31. Tableau • Recall: A basis is feasible basis if . • A feasible basis induces a feasible solution defined by ,. • [Prop 2] If all the coefficients of in are ≤ 0, then the induced is optimal. • Proof: feasible solution : and . Let , then by Prop 1, satisfies . So// // ,

32. Updating… • When updating a tableau, we move a variable from to , then move a variable from to . • The set of variables in allowed to join is: • If :the induced is optimal (by Prop 2). Output it. • The set of variables in allowed to leave is: • If , then the LP is unbounded, because gets increased with as much as we want.

33. Updating • The updating rule maintains the tableaus: • Theorem. , ,

34. Pivoting rule: which in (and which in ) to pick? • Largest coefficient in (2). • Dantzig’s original. • Largest increase of z. • Steepest edge: i.e. closest to the vector c. • Champion in practice. • Bland’s rule: smallest index. • Prevents cycling. • Random: • Best provable bounds.

35. Picking the initial feasible solution • [Fact] s.t. and the following LP has optimal value 0 • The new LP has variables . • Proof. : opt . achieves 0.: Take . opt , , . So and .

36. Solve the new LP first • Note that the new LP has a feasible basis easily found: . • . • Solve this new LP first, obtaining an optimal solution together with a feasible solution. • If optimal value ≠ 0: the original LP is not feasible. • If optimal value = 0: we get a basic feasible solution for the new LP, w/ . • Columns of in are linearly independent. Expand it to linearly indep. columns in . • This gives a feasible basis for the original LP.

37. Simplex Alg: putting everything together • If no feasible basis is available, • solve • If optimal value ≠ 0: original LP is infeasible. • If optimal value = 0: get a feasible basis for the original LP.

38. Simplex Algorithm: continued • For the feasible basis , compute tableau • if all coefficients of in are ≤ • output optimal solution , with in , and .(opt value: .) • else • pick by some pivoting rule. • ifthe column of in tableau ≥ 0, output “LP is unbounded”. • else • Pick by some pivoting rule • and go to the first step in this slide.

39. Efficiency • In practice: Very efficient. • Typical: pivoting steps. • : number of constraints • In theory: • Finite: Some pivoting rules prevent cycling. • Worst case complexity is exponential for most known deterministic pivoting rules. • No “pivoting rule”, deterministic or randomized, with polynomial worst-case complexity known. • Best bound: with variables and constraints

40. Theory of simplex method • Actually we don’t even know the complexity of best possible pivoting rule. • Hirsch Conj: It’s . • Best upper bound (Kalai-Kleitman): . • Smoothed complexity: For any LP, perturbing its coefficients by small random amounts makes the simplex method (w/ a certain pivoting rule) polynomial time complexity. • See here for surveys/papers.

41. Recall our problem: max s.t. (1) (2) (3) (4) Let’s see how good the solution could be. : It’s an upper bound. It’s a better upper bound. What’s the best upper bound obtained this way? Duality

42. Recall our problem: max s.t. (1) (2) (3) (4) In general: If and , wegetanupperbound: . The best upper bound? min s.t. Duality This is another linear programming problem. --- dual of the original LP.

43. Making it formal • Primal max s.t. • Dualmin s.t.

44. General form of the LP-duality • Primal Dual max min s.t. s.t. max min s.t. s.t. • variable constraint • max min • contraints variables unrestricted • variable constraint

45. Strong duality • The primal gives lower bounds for the dual • The dual gives upper bounds for the primal • [Strong duality] For linear programming, optimal primal value = optimal dual value • If both exist, then they are equal • If one is infinity, then the other is infeasible

46. Application: Zero-sum game • Two players: Row and Column • Payoff matrix • : Row pays to Column when Row takes strategy and Column takes strategy • Row wants to minimize; Column wants to maximize. • Game: You don’t know others’ strategy.

47. Who moves first? • They both want to minimize their loss in the worst case (of the other’s strategy). • Row: • Column: • Fact: • Game theoretical interpretation: The player making the first move has disadvantage. • Consider the Rock-Paper-Scissors game: If you move first, then you’ll lose for sure.

48. Mixed strategy • Mixed strategy: a randomized choice. • Row: strategy with prob. . • Column: strategy with prob. . • Now the tasks are: • Row: • Column: • Fact: the inner opt can be achieved by a deterministic strategy. • So the tasks become: • Row: • Column:

49. Minimax • Minimax theorem: = • The player who moves first doesn’t have disadvantage any more! • Consider the Rock-Paper-Scissors game again: Each player wants to use distribution on her choices.

50. Row: min s.t. Column: max s.t. Proof by LP duality • Observation: These two LP’s are dual to each other. • Thus they have the same optimal value.