1 / 46

Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming

Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming. 15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu.edu Wean Hall 4126. Recap. Convex optimization is a convex function and is a convex set

sandin
Download Presentation

Artificial Intelligence: Representation and Problem Solving Optimization (2): Linear Programming

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Artificial Intelligence: Representation and Problem SolvingOptimization (2): Linear Programming 15-381 / 681 Instructors: Fei Fang (This Lecture) and Dave Touretzky feifang@cmu.edu Wean Hall 4126

  2. Recap • Convex optimization • is a convex function and is a convex set • Gradient descent leads to global optimum s.t. Fei Fang

  3. Recap • Convex optimization • is a convex function and is a convex set • Gradient descent leads to global optimum • Nonconvex but continuous optimization • Gradient descent leads to local optima s.t. Reference: Wikipedia; Berkeley CE191

  4. Recap • Convex optimization • is a convex function and is a convex set • Gradient descent leads to global optimum • Nonconvex but continuous optimization • Gradient descent leads to local optima • This section: is a linear function and is defined by a set of linear constraints s.t. Reference: Wikipedia; Berkeley CE191

  5. Outline • Linear Program • Simplex Algorithm • LP Duality Fei Fang

  6. Linear Program: Definition • Linear Program: • A special case of convex optimization problem • An optimization problem whose optimization objective is a linear function and feasible region is defined by a set of linear constraints s.t. Note: can also be minimization Fei Fang

  7. Linear Program: Example • Example: Maximize Profit in Manufacturing s.t. s.t. Fei Fang

  8. Linear Program: Example • Example: Maximize Profit in Manufacturing s.t. Fei Fang

  9. Quiz 1: Linear Program • Which constraints determine point B? Fei Fang

  10. Linear Program: Definition • Feasible region defined by • define a halfspace • Feasible region is the intersection of halfspaces, i.e., a linear polytope s.t. Fei Fang

  11. Linear Program: Definition • Objective function • Represent a direction of optimization • Generally, optimum always occur on a corner, i.e., a vertex of the feasible region s.t. Fei Fang

  12. Linear Program: Definition • Infeasible LP • Feasible region is empty • Unbounded LP • Unbounded polytope • Optimal value s.t. s.t. Fei Fang

  13. Linear Program: How to Solve • At least one vertex of the polytope is an optimal solution • If is a convex combination of two different feasible points , then the objective value at either or is no worse than the value at Fei Fang

  14. Linear Program: How to Solve • Naïve approach • Find all vertices • Pick the one with the best objective value How? Fei Fang

  15. Linear Program: How to Solve • Find vertex • linearly independent linear equations uniquely determines a point in • Choose inequality constraints and set as equality • Solve the linear system s.t. Fei Fang

  16. Linear Program: How to Solve • Find vertex for LP in standard form • Any LP can be converted into standard form • If maximization, convert to minimization • For any constraint , add slack variables and get s.t. s.t. Fei Fang

  17. Linear Program: How to Solve • Find vertex for LP in standard form Where • Focus on • If rows of are not linearly independent, apply basic row operation to remove redundant constraints. There is no solution if it leads to contradicting constraints. • If : Unique point . No optimization needed. • If : Infinite points satisfying • So we only consider and s.t. Fei Fang

  18. Linear Program: How to Solve • Find vertex of the feasible region ,, equality constraints, inequality constraints • Choose a set of variables, denoted as • Solve the following linear system, get a solution • is a vertex of the feasible region if satisfies remaining constraints , how many choices? Only variables in can take non-zero values denotes subselecting columns of based on , denote subselecting element of Fei Fang

  19. Linear Program: How to Solve Only variables in can take non-zero values • Choose • Solve linear system s.t. Fei Fang

  20. Linear Program: How to Solve Only variables in can take non-zero values • Choose • Solve linear system What if add a constraint ? s.t. Fei Fang

  21. Linear Program: How to Solve s.t. feasible Corresponds to vertex B in x-y space feasible Corresponds to vertex D in x-y space infeasible Fei Fang

  22. Linear Program: How to Solve s.t. infeasible feasible Corresponds to vertex A in x-y space feasible Corresponds to vertex (0,0) in x-y space Fei Fang

  23. Linear Program: How to Solve • Naïve approach • Find all vertices • Pick the one with the best objective value • Practical approach: Simplex algorithm • Start from one vertex • Move towards a neighboring vertex for improvement s.t. Fei Fang

  24. Linear Program: How to Solve • Naïve approach • Find all vertices • Pick the one with the best objective value • Practical approach: Simplex algorithm • Start from one vertex • Move towards a neighboring vertex for improvement • Recall local search and gradient descent • May enumerate almost all the vertices in the worst case, but very efficient in most cases Fei Fang

  25. Simplex Algorithm • Again, focus on standard form • A vertex is the solution of the linear system for a certain s.t. Fei Fang

  26. Simplex Algorithm • A single step in simplex algorithm: Given a vertex and corresponding , move to the neighboring vertex with a decrease in objective value Current solution: A neighboring vertex of only has one different element in Fei Fang

  27. Simplex Algorithm • Consider adjusting toby setting for some while ensuring , and • All has to change accordingly • Denote , then Fei Fang

  28. Simplex Algorithm • The new objective value is • Rewritten as where is called reduced cost • If , then and the adjustment is worthwhile. The larger the , the more improvement per unit of change • To ensure , we have • Let . So . • When , we have and a neighboring vertex is reached Fei Fang

  29. Simplex Algorithm Note: In practice, when checking if and (and other tie breaking conditions), we can compare to to avoid numerical issues Fei Fang

  30. Simplex Algorithm • Example , s.t. • Introduce to • Reduced cost . If increases by , obj value decreases by • . So if increases by , has to decrease by , has to decrease by • can increase by at most , at which point becomes • So the new is Corresponds to vertex D in x-y space Fei Fang

  31. Simplex Algorithm , • Example s.t. • Introduce to • Reduced cost • . So if increases by , has to decrease by , has to decrease by • can increase by at most , at which point becomes • So the new is Corresponds to vertex B in x-y space • At this point, none of the variables has a negative reduced cost , so optimal solution is found Fei Fang

  32. Simplex Algorithm • Handling degeneracy: avoid cycling over the indices • Bland’s rule • At each step, choose smallest 𝑗 such that < 0 • For variables that could exit set choose the smallest Fei Fang

  33. Quiz 2 • Consider the following two LPs (LP-L and LP-R) where • Applying simplex algorithm to LP-L with . Denote the optimal solution as . If , then which of the following claims are true about ? • A: is not in the feasible region of LP-R • B: is in the feasible region of LP-R • C: is a vertex of the feasible region of LP-R • D: is an optimal solution of LP-R LP-R LP-L s.t. s.t. Fei Fang

  34. Two-Phase Simplex Algorithm s.t. • Phase I (Find one vertex): Apply simplex alg to the following LP with initial vertex , get • If , the original LP is infeasible • Phase II (Find best vertex): Apply simplex alg to the original LP with initial vertex s.t. For , flip the sign of the constraints , provides a vertex to begin with Fei Fang

  35. Simplex Algorithm • Theorem: the simplex algorithm is guaranteed to find a globally optimal solution to a linear program Proof: If simplex returns, then it has found a point where we cannot improve the objective locally; since linear programs are convex, this is a global optimum Because the objective of the simplex improves at each iteration, and since there are a finite (but exponential) number of vertices in the polytope, the algorithm must return after a finite number of steps Fei Fang

  36. Linear Program: How to Solve • Naïve approach • Find all vertices • Pick the one with the best objective value • Practical approach: Simplex algorithm • Start from one vertex • Move towards a neighboring vertex for improvement • Recall local search and gradient descent • May enumerate almost all the vertices in the worst case, but very efficient in most cases • (Not covered) Ellipsoid algorithm • Polynomial-time in theory, poor performance in practice Fei Fang

  37. Linear Program: How • Solve LPs in practice: • All the solvers/algorithms for Convex Optimization problems can be applied • Additional solvers/algorithms for LPs • linprog (MATLAB), linprog (in Python package SciPy) • PuLP (Python) • Cplex, Gurobi • https://www.informs.org/ORMS-Today/Public-Articles/June-Volume-38-Number-3/Software-Survey-Linear-Programming Fei Fang

  38. Linear Program: Note • Linear program is a continuous optimization problem, but can be viewed as a combinatorial optimization problem: find a corner of the polytope defined by that maximizes the objective function Fei Fang

  39. Linear Program Duality • Dual problem of an LP: also a linear program • Each dual variable corresponds to a constraint in primal LP • Strong duality theorem: LP and its dual have the same optimal objective value (if feasible and bounded) Primal LP in Standard Form variables, constraints Dual LP variables, constraints s.t. s.t. , Fei Fang

  40. Linear Program Duality Primal LP in Standard Form Dual LP s.t. s.t. s.t. s.t. Fei Fang

  41. Linear Program Duality s.t. s.t. Optimal obj Fei Fang

  42. Write the Dual of an LP (Non-Standard form) s.t. s.t. Fei Fang

  43. Other properties of LP Duality (not required) • Complementary slackness • Primal constraint is not tight dual variable= • Primal variable is not zero dual constraint is tight • Dual constraint is not tight primal variable= • Dual variable is not zero primal constraint is tight • Optimality Conditions: If are feasible solutions to the primal and dual problems, respectively, then they are optimal solutions to these problems if, and only if, the complementary-slackness conditions hold for both the primal and the dual problems Fei Fang

  44. Summary Optimization Problems Convex Programs Gradient Descent Linear Programs Simplex Fei Fang

  45. Linear Program: Additional Resources • Textbook • Applied Mathematical Programming, Chapters 2-4 • By Bradley, Hax, and Magnanti (Addison-Wesley, 1977)  • http://web.mit.edu/15.053/www/AMP.htm • Online course • https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-251j-introduction-to-mathematical-programming-fall-2009/index.htm Fei Fang

  46. Acknowledgment • Some slides are borrowed from previous slides made by J. Zico Kolter Fei Fang

More Related