Linear programming Thomas S. Ferguson University of California at Los Angeles

1. Linear programmingThomas S. FergusonUniversity of California at Los Angeles Compressive Sensing Tutorial PART 3 Svetlana Avramov-Zamurovic January 29, 2009.

2. Definitions • A linear programming: problem of maximizing or minimizing a linear function subject to linear constraints. • xi≥ 0: nonnegativity constraints. • Ax≤b: main constraints (inequalities). • A vector x or y feasible if it satisfies the corresponding constraints. • Constraint set is set of feasible vectors. • LP problem is feasible if constraint set is not empty. • The function to be maximized (or minimized) is called the objective function cTx. • Objective function always takes on its maximum (minimum) value at a corner point of the constraint set if the constraint set is bounded. • A feasible problem is unbounded if the objective function can assume arbitrarily large values at feasible vectors. Otherwise it is bounded. • Some constraints may be equalities. • Solve the equality, eliminate the variable. • (b) Some variables may not be • restricted to nonnegatives. • x=u-v (u,v are nonnegative, u>0, v>0). • This adds extra variable to the problem

3. DualityMAX cTx, due to Ax≤b, x>0MIN yTb, due to yT A≥cT, y>0 MAX variable MIN variables Constraints Objective function

4. Duality: If you conjecture (guess) a solution to one problem, you may solve the dual using the complementary slackness conditions and than see if your conjecture is correct. • Thm 1: If x is feasible for the standard MAX problem and if y is feasible for its dual then cTx ≤ yTb. • C1: If a standard problem and its dual are both feasible, then both are bounded feasible. • C2: If there exists feasible x* and y* for a standard MAX problem and its dual such that cTx* = y* Tb than both are optimal for their respective problems. • The Duality Thm: If a standard LP problem is bounded feasible, than so its dual, their values are equal, and there exist optimal vectors for both problems. • Equilibrium Thm: Let x* and y* be feasible vectors for a standard MAX problem and its dual respectively. Then x* and y* are optimal if and only if Complementary slackness conditions: strict inequality (slackness) in a standard problem constraint implies that complementary constraint in the dual be satisfied with equality

5. The Pivot Operation Pivoting results in matrix inverse (special case) yT A= sT => yT =A-1 sT

6. Linear programming A system of linear inequalities defines a polytope as a feasible region. The simplex algorithm begins at a starting vertex and moves along the edges of the polytope until it reaches the vertex of the optimum solution.

7. LP MIN: Find y to minimize yTb subject to y≥0 and yTA ≥ cTSimplex methodFind y and s to minimize yTb subject to y≥0, s≥0 and sT = yTA-cT Slack variable MIN variables Objective function Simplex TableausT = yTA-cT

8. Simplex methodis pivot madly method that tells you which points to choose as pivots to approach the solution systematically. • If -c≥0 and b≥0, there is an obvious solution to the problem: the minimum occurs at y=0; and s=-c and the minimum value yTb=0. • Assume: there is at least one nonnegative entry in the last column or last row (exclusive of the corners). If we pivot about a11, including the last column and the last row we have r=(y1, s2,…, sn) and t=(s1,y2, …,ym) • Find vectors y and s to minimize tTb subject to y≥0, s≥0 and r=tTA^-c^ • The Pivot Madly method: Pivot madly until you suddenly find -c^≥0 and b^≥0 Then, obtain a feasible point by setting r=-c and t=0 The value is the lower right corner. LP Max problem (slack variable u=b-Ax, same tableau if –u=Ax-b)

9. Pivot rules for simplex methodfrom the point of the MAX problem • b≥0. Take any column with last entry negative, say -cj0<0. Among those i for which ai,j0>0, choose that i0 for which the ratio bj/ai,j0 is smallest. If there are ties, choose any such i0.Pivot around ai0,j0. • After pivoting the b column stays nonnegative, so you still have a feasible point for the maximum problem. • The value of the new tableau is never less than the old (generally greater.) • b^i=bj-ai,j0bj0/ ai0,j0 and b^i=bj0/ ai0,j0 • v^=v-(cj0)bj0/ ai0,j0 • Some bi are negative. Take the first negative bj, say bk<0 (where b1, … bk-1≥0). Find any negative entry in row k, ak,j0<0. The pivot will be column j0. Compare bk/ak,j0 and bj/ai,j0 for which bj>0 and ai,j0>0 and choose i0 for which this ratio is smallest (i0 may be equal to k). You may choose any such i0 if there are ties. Pivot on ai0,j0 . George Dantzig in 1947 The algorithm moves along the edges of the polyhedron defined by the constraints, from one vertex to another, while decreasing the value of the objective function, cTx at each step.

10. Pivot rules for simplex methodfrom the point of the MIN problem • -c≥0. Take any column with last entry negative, say bi0<0. Among those j for which ai0,j<0, choose that j0 for which the ratio -cj/ai0,j is closest to 0. If there are ties, choose any such j0.Pivot around ai0,j0. • Some -ci are negative. Take the first negative -ci , say -ck<0 (where -c1, … -ck-1≥0). Find any positive entry in coulmn k, ai0,k>0. Compare -ck/ai0,k and -cj/ai0,j for which -cj>0 and ai0,j<0 and choose j0 for which this ratio is clossest to zero (j0 may be equal to k). Pivot on ai0,j0 .