University of Massachusetts at Amherst 665 Computer Algorithm Mathematical Programming Professor Maciej Ciesielski Spri

1. University of Massachusetts at Amherst 665 Computer Algorithm  Mathematical Programming Professor Maciej Ciesielski Spring 2004 (DFG)

2. OUTLINE • Non Linear Programming • Quadratic Programming • Linear Programming • Geometric Solutions • Basic Solutions • Simplex Method • LP modifications • Integer Programming • Unimodularity • Network Flow Problems • Conclusions

3. Mathematical Programming Nonlinear Programming Minimize f(x) subject to: g(x) = 0 h(x) ≥ 0 where x = [x1,...xn]T, and all functions f, g, h, are differentiable. {

4. Convexity  To obtain global optimum, it is important that the constraint set {x | g(x) = 0,h(x) ≥ 0}  and the function f(x) be convex. If the functionis convex over a convex constraint set, then local minimun is also a globa1 minimum

5. Set convexity Convex set non-convex set Function convexity Convex function non-convex function

6. Classical optimization deals with unconstrained optimization problems. • Solution: move the constraints to the function and solve the unconstrained optimization problem • For the minimization problem • Min f(x) subject to g(x) = 0 • Penalty function method: replace f(x) by • F(x) = f(x)+ M( g(x))2, M >> 0 • 2. Lagrangian multipliers: replace f(x) by • L(x,λ) = f(x) + Σ λi gi(x) • At the optimum point, x*, L(x*, λ) = f(x*) i

7. Quadratic Programming (QP) Minimize xTQx Subject to: {Ax ≥ b Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector Matrix Q is positive definite, i.e., xTQx > 0 For every x  0. Thus f is convex.

8. Linear Programming (LP) Minimize cT Subject to Ax ≥ b x ≥ 0 Where cT is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector. The constraint set {x | Ax ≥ b} Is a convex polyhedron, and f is linear, convex. {

9. Linear Programming (LP)Optimization problem:• 2 types of products are made by the factory• 3 machines are needed to make each productConstraints: Machines are available for a limited time during one production session:• time on A ≤50 hours • time on B ≤35 hours • time on C ≤ 80 hours

10. To complete the product, each lot must be processed by all three machines for a certain number of hours Required manufacturing time Machines A B C 1 2 Time Constraint: 50 35 80

11. Profit •   Product 1 can be sold for $100 per lot • • Product 2 can be sold for $80 per lot • Optimization Problem: • Design the production schedule which rnaximizes the profit of both products manufactured during one production session.

12. Formulation x1 = number of lots of product 1 x2 = number of lots of product 2 max f (x) = 100x1 + 80x2 10x1 + 5x2≤ 50 5x1 +   5x2≤35 5x1 +   15x2≤ 80 {

13.  Convert the problem to a standard Linear Program (LP): Max f(x) = 100x1 + 8Ox2 10x1+ 5x2+ s1 = 50   s. to 5x1 + 5x2+ s2 = 35 5x1+ 15x2+ s3 = 80 Basis New Matrix A {

14. Geometric solutionFamily of curves, constant for x1,x2,F(x) = 100 x1, +80 x2 =800 (const)(a) 10x1, +5x2 50 (b) 5x1, + 5x2 35(c) 5x1, + 15x2 80 x2 15 10 5 4 P2 P1 P0 c 3 5 10 15 F(x) = 800 x1 a b

15. Basic Solution Given a system of equalities Ax=b where x = n - vector, b = m - vector, A = mxn matrix, select a set of m linearly independent columns such that (mxm) matrix B is nonsingular, i.e. /B / 0. Then one may uniquely solve the equation Bxb= b where Xbis an m - element subvector of x.

16. By putting x=(xb,0) we obtain a solution to Ax = b Such a solution (with n - m components of xnot associated with columns of B m x m ) to theresulting set of equations is said to be a basic solution with regard to the basis B. The Xbvariables, associated with columns of B, are called basic variables. Basic Feasible Solution If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution.

17. Fundamental Theorem of Linear Programming Given an L.P. in standard form min cTx Ax = b x  0 where A is m x n matrix of rank m (i.e. m < n and the m rows are linear independent), s. to • If there is a feasible solution, there is a basic feasible solution • If there is an optimal feasible solution, there is an optimal basic feasible solution

18. This theorem shows that it is necessary only to consider basic feasible solutions when seeking an optimal solution to a linear program, as the optimal value is always achieved as such a solution Ultimate goal: find a basic feasible solution with a base B composed of original variables only, and which is optimum.

19. ƒ (x) = 100 x1, +80 x2 +0 s1 +0 s2 +0 s3 x1 x2 s1 s2, s3 10 5 100 5 5 010 5 15 001 5 0 3 5 8 0 = basis Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is a basic feasible solution (point P0 (0,0)) Function value:F(0, 0. 50, 35, 80) = 0

20. Simplex Method Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually decrease the value of the f (x) until a minimum is reached.  General comments It is easy to find initial basic feasible solutions with slack variables. Finding initial basic solution is part of the Simplex method (Lue 84).

21. Simplex Algorithm • Select the column, such that the new resulting basic feasible solution will yield a lower valueto f(x) than the previous one. • Select the pivot element in that column. By an elementary evaluation determine • Which vector ajshould enter the basis (so that f(x) is reduced), and • Which vector should leave the basis.

22. Construct a Simplex Tableau, [A, b ] appended by a row at the top with cost coefficients and current cost. Point P0 (0,0) F(0,0,50,35,80) = 0

23. Step 1. Select column q with rq > 0. (xqenters the basis) q : rq = max ri Step 2. Select pivot element p in column q Which minimizes ratio: p : = min for positive yiq yi0 yp0 yi0 yiq ypq yiq i

24. yi q yp q • ypj , j = 0, 1,…, n Step 3. Pivot on element ypq and update the tableau (update all rows, including the top one) • In row p divide ypjby ypq, j = 0, 1,…, n • Subtract from each row i  p

25. Pivot on y1 1 Point P0 (0,0) F(0,0,50,35,80) = 0 Update the table Point P1 (5,0) F(5,0,0,10,55) = 500

26. Pivot on y2 2 Point P1 (5,0) F(5,0,0,10,55) = 500 Update the table Point P1 (3,4) F(3,4,0,0,5) = 620

27. LP Modifications 1. Maximization max c t x = min (-c t x) 2. Unconstrained x let x = x+ - x -; x+, x - 0) min (c t x+ - c t x -) x+ x - [A, -A] = b s. to x+, x - 0

28. x xs min [cT , 0] x xs 3. Constraint set:Ax  b Slack variables,xs : Ax + xs = b [A, I ] = b x , xs  0 Columns corresponding to xsform a basic solution

29. x xp min [cT , 0] x xp 4. Constraint set:Ax  b Surplus variables,xp : Ax - xp = b [A, -I ] = b x , xp  0

30. x xp xa x xp xa Artificial variables,xa: Ax - xp + xa = b (cannot star with negative basis ofxp ) min [cT , 0 , k I ] [A, -I , I ] = b S. to x , xp ,xa 0 Columns corresponding to xaform a basic solution

31. Integer Programming Integer Program max cTx Ax  b xiinteger Typically, xi = 0,1 (0,1 Integer Programming) s. to

32. 2 e1 e2 1 3 e3 IP Example: matching problem Given a graph G , find maximal set of edges in G, such that no two edges are adjacent to the same vertex. • Maximum matching: matching of maximum cardinality. • Weighted matching: matching with

33. 2 e1 e2 1 3 e3

34. In matrix form: Where b = [ 1,…1]T , A = incidence matrix of G 1 2 3

35. max 1x1 + 1x2 + 1x3 x1 , x2 , x3 = 0, 1 One solution to this IP problem: x1= 1 , x2= 0 , x3= 0 Other possible solutions: x1= 0 , x2= 1 , x3= 0, or x1= 0 , x2= 0 , x3= 1

36. Question Can the solution to this IP problem be obtained by dropping the integrality constraint: xi = 0, 1 And solving the LP problem instead? In our example, solution to the IP is not obtainable from LP. Reason:matrix A does not have certain property (total unimodularity) needed to guarantee integer solutions.

37. If xi = 0, 1is relaxed, such that xi 0 ,then the solution to the associated LP problem is non – integer: Reason: the A matrix Is not totally unimodular: |A| = - 2

38. Unimodularity: LP -> IP Given a constraint set in standard form where A, b are integer Partition A = [B/N ]; x = [xB, xN ] B is nonsingular m x m basis, N is non-basic

39. Basic solution is In particular, when B = I and B -1 = I then XB= b a solution can be obtained by inspection (as in the initial step of Simplex method).

40. Since xB = B –1 bwith xN = 0, binteger A sufficient condition for a basic solution xBto be integer is that B –1 be an integer matrix

41. Unimodularity A square matrix B is called unimodular if D = |det B| = 1 An integer matrix A is totally unimodular if every square, nonsingular submatrix of A is unimodular. Equivalently: A is totally unimodular if every subdeterminant of A is 0, +1, or –1.

42. Recall that for B nonsingular: Where B + , adjoint matrix = [ i j cofactor of element a i jin det A ]T B +and det B are integer if B –1 is integer

43. Cofactor of ai j : Determinant obtained by omitting the ith row and the jth column of A and then multiplying by (-1) i + j .

44. For B unimodular, B –1integer: If A is totally unimodular, every basis matrix B is unimodular and every basic solution ( xB , yN) = ( B –1 b, 0 ) Is integer. In particular, the optimal solution is integer

45. Theorem 1: If A is totally unimodular then every basic solution of Ax = b is integer. For LP’s with equality constraints total unimodularity is sufficient but not necessary. For LP with inequality constraints, Ax  b, total unimodularity of A is both necessary and sufficient for all extreme points of s = {x : Ax  b , x  0 } to be integer for every integer vector b.

46. Conclusion: Any IP with totally unimodular constraint matrix can be solved as an LP. Totally unimodular matrix : a i j = 0 , +1, -1 Also, every determinant of A must be 0, +1 or -1

47. Special Linear Programs with Integer Solutions Network flow problems: • max flow • min – cost flow • assignment problem • shortest path • transportation problem are LP with the property that they possess optimal solutions in integers.

48. Some Totally Unimodular Linear Programs Assignment problem (special case of min – cost capacitated flow problems) [ m jobs x m men ] c i j =cost of assigning man i to job j

49. s. to.

50. X11 X12 X13 X21 X22 X23 X31 X32 X33