1 / 73

Introduction to optimization

Introduction to optimization. Dr. Md. Golam Hossain Professor Department of Statistics University of Rajshahi Rajshahi-6205, Bangladesh E-mail:hossain95@yahoo.com HP: +8801914254013. Module Outline. Mathematical Concepts for Optimization Introduction to Linear Programming

hart
Download Presentation

Introduction to optimization

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to optimization Dr. Md. Golam Hossain Professor Department of Statistics University of Rajshahi Rajshahi-6205, Bangladesh E-mail:hossain95@yahoo.com HP: +8801914254013

  2. Module Outline • Mathematical Concepts for Optimization • Introduction to Linear Programming • Graphical Method • Simplex Method for Maximization (only type of constraints “≤”) Problem • Simplex Method for Minimization or Maximization (Type of constraints ≥ or mix) Problem • Duality in Linear Programming

  3. Mathematical Concepts for Optimization • In Mathematics, Statistics, Empirical Sciences, Computer Science or Management Science, Mathematical optimization (Alternatively, Optimization or Mathematical Programming) is the selection of a best element (with regard to some criteria) from some set of available alternatives. • More generally, optimization includes finding "best available" values of some objective function given a defined domain, including a variety of different types of objective functions and different types of domains.

  4. Optimization Problem An optimization problem can be represented in the following way: A function f : A   R from some set A to the real numbers. An element x0 in A such that f(x0) ≤ f(x) for all x in A (minimization) or such that f(x0) ≥ f(x) for all x in A (maximization). Such a formulation is called an optimization problem or a mathematical programming problem.  

  5. We say that the function f (x) has a relative maximum value at x = a, if f (a) is greater than any value in its immediate neighborhood. We also say that a function f (x) has a relative minimum value at x = b, if f (b) is less than any value in its immediate neighborhood. The value of the function, the value of y, at either a maximum or a minimum is called an extreme value.

  6. Sufficient conditions We can now state these sufficient conditions for extreme values of a function at a critical value a: The function has a minimum value at x = a  if f '(a) = 0 and f ''(a) = a positive number. The function has a maximum value at x = a  if f '(a) = 0 and f ''(a) = a negative number. Example.  Let  f(x) = x² − 6x + 5.   Are there any critical values -- any turning points?  If so, do they determine a maximum or a minimum?  And what are the coordinates on the graph of that maximum or minimum?

  7. History of Optimization Greek mathematicians solved some optimization problems that were related to their geometrical studies • Euclid [Father of Geometry] • Residence: Alexandria, Egypt, Field: Mathematics, Known for: Euclidean geometry, Euclid's Elements. • Contribution:300 BC Euclid proved that a square has the greatest area among the rectangles with given total length of edges. • Heron (100 BC) • Residence: Alexandria, Roman Egypt, Field: Mathematics. • Contribution: 100 BC Heron proved that light travels between two points through the path with shortest length when reflecting from a mirror.

  8. 17th and 18th centuriesBefore the invention of calculus of variations (calculus) only some separate optimization problems were being investigated. Pierre de Fermat (France): [1601-1665] Field: Mathematics and Law. Contribution: In 1646 P. de Fermat showed that light travels between two points in minimal time. Johannes Kepler (Germany) [1571-1630] Field: Astronomy, Astrology, Mathematics and Natural Philosophy Contribution: In 1615 he Calculated the optimal dimension of wine barrel.

  9. I. Newton(1660s) and G. W. von Leibniz(1670s) create mathematical analysis that forms the basis of calculus. Some separate finite optimization problems are also considered. Isaac Newton (England): [1642-1727] Field: Physics, Mathematics, Astronomy, Natural Philosophy,  Alchemy, Christian Theology. Contribution: In 1687 Newton studied the body of minimal resistance. Gottfried Wilhelm Leibniz (German): [1646-1716] Field: Mathematics, Metaphysics, Theodicy. Contribution: In 1670 he studied some separate finite optimization problems in calculus.

  10. Gaspard Monge (France): [1746-1818]. Field: Mathematics and Engineering Education. Contribution: In 1784 G. Monge investigated a combinational optimization problem known as the transportation problem. Leonhard Euler (Switzerland): [1707-1983] Field: Mathematics and Physics. Contribution: In 1740 he was first researcher who published research on general theory of calculus. Joseph Louis Lagrange (France): [1736-1813]. Field: Mathematics and Mathematical Physics. Contribution: In 1760 he formulated the Plateau’s problem, the problem of minimal surfaces.

  11. 19th century The first optimization algorithms are presented. K.T.W. Weierstrass, J. Steiner, W.R. Hamilton and C.G.J. Jacobi further develop of calculus of variations. Carl Gustav Jacob Jacobi (German): [ 1804- 1851] Field: Mathematics William Rowan Hamilton (Irish): [1805–1865]. Field: Physics, Astronomy and Mathematics. Karl Theodor Wilhelm Weierstrass (German): [1815-1897]. Field: Mathematics.

  12. Jean Baptiste Joseph Fourier (France): [1768-1830]. Fields: Mathematics, Physics and History. Contribution: In 1826 J.B.J. Fourier formulated LP-problem for solving problems arising in mechanics and probability theory. Léon Walras (France) : [1834-1910]. Field: Economics, Marginalism. Contribution: In 1870 Walras and Cournot shifted the focus of economists to utility maximizing individuals optimization becomes an integral part of economic theory.

  13. 20th Century Leonid Kantorovich (Soviet Russia): [1912-1986] Field: Mathematics. Contribution: 1939 L.V. Kantorovich presented LP-model and an algorithm for solving it. In 1975 Kantorovich and T.C. Koopmans got the Nobel memorial price of economics for their contributions on LP-problem. Harris Hancock (America): [1867-1944] Field: Mathematics. Contribution: H Hancock published the first text book on optimization, Theory of Maxima and Minima in 1917. Johan Ludwig William Valdemar Jensen (Denmark): [1859-1925] Field: Mathematics. Contribution: He introduced Convex functions in 1905.

  14. After the world War II optimization develops simultaneously with operations research. J. Von Neumann is an important person behind the development of operations research. The field of algorithmic research expands as electronic calculation develops. George Bernard Dantzig (America): [1914 – 2005] Field: Mathematics, Operation Research, Computer Science, Economics and Statistics Contribution: In 1947 G.B.Dantzig designed the “simplex method” for solving linear programming formulations of U.S. Air Force planning problems. John von Neumann (United States): [1903-1957] Field: Mathematics and Computer Science Contribution: Additionally he established the theory of duality for LP-problems 1949 the first international congress, International Symposium on Mathematical Programming, on optimization is held in Chicago. The number of papers presented in the congress was 34.

  15. Linear Programming Linear programming has nothing to do with computer programming. The use of the word “programming” here means “choosing a course of action.” Linear programming involves choosing a course of action when the mathematical model of the problem contains only linear functions. 15

  16. Historical Perspective for LP • 1928 – John von Neumann published related central theorem of game theory • 1944 – Von Neumann and Morgenstern published Theory of Games and Economic Behavior • 1936 – W.W. Leontief published "Quantitative Input and Output Relations in the Economic Systems of the US" which was a linear model without objective function. • 1939 – Kantoravich (Russia) actually formulated and solved a LP problem • 1941 – Hitchcock poses transportation problem (special LP) • WWII – Allied forces formulate and solve several LP problems related to military • A breakthrough occurred in 1947...

  17. SCOOP • US Air Force wanted to investigate the feasibility of applying mathematical techniques to military budgeting and planning. • George Dantzig had proposed that interrelations between activities of a large organization can be viewed as a LP model and that the optimal program (solution) can be obtained by minimizing a (single) linear objective function. • Air Force initiated project SCOOP (Scientific Computing of Optimum Programs) NOTE: • SCOOP began in June 1947 and at the end of the same summer, Dantzig and associates had developed: • 1)An initial mathematical model of the general linear programming problem. • 2)A general method of solution called the simplex method.

  18. Requirements of linear Programming • In general, linear programming can be used for optimization problems if the following conditions are satisfied: • There must be a well-defined objective function (profit, cost or quantities produced) which is to be either maximized or minimized and which can be expressed as a linear function of decision variables. • There must be restrictions on the amount or extent of attainment of the objective and these restrictions must be capable of being expressed as linear equalities or inequalities in terms of variables. • There must be alternative course of action. For example, a given product may be processed by two different machines and the problem may be as to how much of the product to allocate to which machine. • Another necessary requirement is that decision variables should be interrelated and non-negative. The non-negativity condition shows that linear programming deals with real-life situations for which negative quantities are generally illogical. • The recourses must be in limited supply. For example, if a firm starts producing greater number of a particular product, it must make smaller number of other products as the total production capacity is limited.

  19. Linear Programming (LP) Problem • If both the objective function and the constraints are linear, the problem is referred to as a linear programming problem. • Linear functions are functions in which each variable appears in a separate term raised to the first power and is multiplied by a constant (which could be 0). • Linear constraints are linear functions that are restricted to be "less than or equal to", "equal to", or "greater than or equal to" a constant.

  20. Guidelines for Model Formulation • Understand the problem thoroughly. • Describe the objective. • Describe each constraint. • Define the decision variables. • Write the objective in terms of the decision variables. • Write the constraints in terms of the decision variables.

  21. Example1: Iron Works, Inc. seeks to maximize profit by making two products from steel. It just received this month's allocation of 19 pounds of steel. It takes 2 pounds of steel to make a unit of product 1, and 3 pounds of steel to make a unit of product 2. The physical plant has the capacity to make at most 6 units of product 1, and at most 8 units of total product (product 1 plus product 2). Product 1 has unit profit 5, and product 2 has unit profit 7. Formulate the linear program to maximize profit. • Answer: • Here is a mathematical formulation of the objective. • Let x1 and x2 denote this month's production level of • product 1 and product 2. • The total monthly profit = (profit per unit of product 1) x (monthly production of product 1) + (profit per unit of product 2) x (monthly production of product 2) • = 5x1 + 7x2 • Maximize total monthly profit: Max z = 5x1 + 7x2

  22. Here is a mathematical formulation of constraints. The total amount of steel used during monthly production = (steel used per unit of product 1) x (monthly production of product 1) + (steel used per unit of product 2) x (monthly production of product 2) = 2x1 + 3x2 That quantity must be less than or equal to the allocated 19 pounds of steel (the inequality < in the constraint below assumes excess steel can be freely disposed; if disposal is impossible, then use equality =) 2x1 + 3x2< 19 The constraint that the physical plant has the capacity to make at most 6 units of product 1 is formulated x1< 6 The constraint that the physical plant has the capacity to make at most 8 units of total product (product 1 plus product 2) is x1 + x2< 8

  23. Adding the non-negativity of production completes the formulation Objective Function Max Z= 5x1 + 7x2 s.t. x1< 6 2x1 + 3x2< 19 x1 + x2< 8 x1> 0 and x2> 0 “Regular” Constraints Non-negativity Constraints “Max” means maximize, and “s.t.” means subject to.

  24. Linear Programming Problem can be solved by • Graphical Method Isoprofit line solution method Corner point solution method • Simplex Method Simplex Method II (Minimization or maximization for constraints, ≥ or mix) Simplex Method I (Maximization, for constrains, ≤)

  25. Some important mathematical items related to method solving LPP • Hyperplane in Rn: The set of points x =[ x1, x2,…,xn] satisfying the equation, z = c1x1+c2x2+...+cnxn, where ci’s are constants not all zero, and z is a constant, is called a hyperplane, determined by the vector c = (c1, c2, …, cn) and scalar z. • Half Spaces: In Rn, let c.x =z be a hyperplane, then the two sets of points • K+= {x|c.x≥z} and K- = {x|c.x≤z} are called the closed half spaces, determined by c.x=z. • Convex Set: A subset S C Rn is said to be convex, if for each pair of points x, y in S, the line segment [x:y] is contained in S.

  26. Some Basic Definitions: • Objective Function: The linear function, z = c1x1+c2x2+…+cnxn, which is to be minimized or maximized is called the objective function of the GLPP. • Subject to the Constraints ( Linear constraints): are linear functions that are restricted to be "less than or equal to", "equal to", or "greater than or equal to" a constant. • Thus 2x1 + 3x2< 19 is a linear constraint, but • 2x1 + 3x2 < 19 and 2x1 + 3x1x2< 19 are not. • Solution: Values of decision variables xj (j=1, 2, 3,…,n) which satisfy subject to the constraints of a GLLP is called the solution to that LPP. • Feasible solution: Any solution to a GLPP which satisfies the non-negative restrictions of the problem, is called a feasible solution to that problem. • Optimum solution: Any feasible solution which optimizes (minimizes or maximizes) the objective function of a GLPP is called an optimum solution to that problem.

  27. Example 1: Graphical Solution x2 • First Constraint Graphed 8 7 6 5 4 3 2 1 x1 = 6 Shaded region contains all feasible points for this constraint (6, 0) x1 1 2 3 4 5 6 7 8 9 10

  28. Example 1: Graphical Solution x2 • Second Constraint Graphed 8 7 6 5 4 3 2 1 (0, 61/3) 2x1 + 3x2 = 19 Shaded region contains all feasible points for this constraint (91/2, 0) x1 1 2 3 4 5 6 7 8 9 10

  29. Example 1: Graphical Solution x2 • Third Constraint Graphed (0, 8) 8 7 6 5 4 3 2 1 x1 + x2 = 8 Shaded region contains all feasible points for this constraint (8, 0) x1 1 2 3 4 5 6 7 8 9 10

  30. Example 1: Graphical Solution x2 • Combined-Constraint Graph Showing Feasible Region x1 + x2 = 8 8 7 6 5 4 3 2 1 x1 = 6 2x1 + 3x2 = 19 Feasible Region x1 1 2 3 4 5 6 7 8 9 10

  31. Example 1: Graphical Solution x2 • Objective Function Line 8 7 6 5 4 3 2 1 (0, 5) Objective Function 5x1 + 7x2 = 35 (7, 0) x1 1 2 3 4 5 6 7 8 9 10

  32. Example 1: Graphical Solution • Selected Objective Function Lines x2 8 7 6 5 4 3 2 1 5x1 + 7x2 = 35 5x1 + 7x2 = 39 5x1 + 7x2 = 42 x1 1 2 3 4 5 6 7 8 9 10

  33. Example 1: Graphical Solution x2 • Optimal Solution Maximum Objective Function Line 5x1 + 7x2 = 46 8 7 6 5 4 3 2 1 Optimal Solution (x1 = 5, x2 = 3) x1 1 2 3 4 5 6 7 8 9 10 Feasible /Convex Region

  34. Problem. A firm manufactures pain relieving pills in two sizes A and B, size A contains 4 grains of element a, 7 grains of element b and 2 grains of element c, size B contains 2 grains of element a, 10 grains of element b and 8 grains of c. It is found by users that it requires at least 12 grains of element a, 74 grains of element b and 24 grains of element c to provide immediate relief. It is required to determine that least no. of pills a patient should take to get immediate relief. Formulate the problem as standard LPP.

  35. Graphical Solution Algorithm • Formulate the given linear programming problem in mathematical format. • Construct the graph for the problem so formulated. • Identify the convex region (solution space) and the vertices of the convex region. • Evaluate the values of the objective function at these vertices of the convex region. • Determine the vertex at which the objective function attains its maximum or minimum (optimum). • Interpret the optimum solution so obtained.

  36. Slack and Surplus Variables • A linear program in which all the variables are non-negative and all the constraints are equalities is said to be in standard form. • Standard form is attained by adding slack variables to "less than or equal to" constraints, and by subtracting surplus variables from "greater than or equal to" constraints. • Slack and surplus variables represent the difference between the left and right sides of the constraints. • Slack and surplus variables have objective function coefficients equal to 0.

  37. Slack Variables (for < constraints) • Example 1 in Standard Form Max Z= 5x1 + 7x2 + 0s1 + 0s2 + 0s3 s.t. x1 + s1 = 6 2x1 + 3x2 + s2 = 19 x1 + x2+ s3 = 8 x1, x2 , s1 , s2 , s3> 0 s1 , s2 , and s3 are slack variables

  38. Surplus Variables (for ≥ constraints) Write down the following LPP in standard form: Minimize Z = 3x1+ 2x2+5x3 Subject to the constraints: 2x1+3x2≥3 x1+2x2+3x3 ≥10 2x1+3x3≥8 x1, x2, x3 ≥0 The Standard Form LPP is, Minimize Z = 3x1+ 2x2+5x3+0.s1 +0.s2+0.s3 Subject to the constraints: 2x1+3x2 –s1 =3 x1+2x2+3x3 –s2 =10 2x1+3x3 –s3 =8 x1, x2, x3, s1, s2, s3≥0 Where, s1 , s2 , and s3 are surplus variables

  39. Some important definitions Standard form of LPP: Let x, c ЄRn and z= cTx be a linear function on Rn. Let A be an m x n real matrix of rank m. Then, the problem of determining x, so as to Maximize z = cTx, subject to the constraints: Ax = b, x≥0, where b is an m x 1 real matrix, is said to be a linear programming problem written in its standard form. Basic Solution: Given a system of m simultaneous linear equations in n unknowns Ax = b; x ЄRn, where A is an m x n real matrix of rank m (m<n). Let B be m x m sub matrix, formed by m linearly independent columns of A. Then a solution obtained by setting n-m variables not associated with the columns of B equal to zero, and solving the resulting system, is called a Basic solution to the given system of equations. The m variables, which may all be different from zero, are called Basic variables. The m x m non-singular submatrix B is called Basis matrix and the columns of B as basic vectors.

  40. Degenerate Solution: A basic solution to the system Ax = b is called Degenerate if one or more of the basic variables vanish. Basic feasible solution: A feasible solution to the LPP which is also basic solution to the problem, is called a basic feasible solution to the LPP. Associated cost vector: Let xB be a basic feasible solution to the LPP: Maximize z= cTx, Subject to constraints: Ax =b, x ≥0. Then the vector cB = [ cB1, cB2, …, cBn], where cBi are components of c associated with the basic variables, is called cost vector associated with the basic feasible solution xB. Optimum Basic Feasible Solution: A basic feasible solution xB to the LPP. Maximize z = cTx, subject to the constraints: Ax =b, x ≥0, is called an optimum basic feasible solution if z0= cBT xB≥z*, where z* is the value of the objective function for any feasible solution.

  41. Example: Find the basic feasible solution to the system of equations: 2x1 +x2 -x3 = 2 3x1 + 2x2 +x3 = 3 Solution. The given system of equations can be written as Ax =b where, A= x = and b= . Since rank of A is 2, the maximum number of linearly independent columns of A is 2. 2 X 2 sub matrices of A are; The variables not associated with the columns of these sub matrices are, respectively, x3, x1 and x2. and Considering B= , a basic solution to the given system is obtained by setting x3= 0 and solving the system Thus a basic solution to the given problem is, (Basic) x1 = 1, x2 = 0; (Non-basic) x3= 0. Similarly the other two solutions are, (Basic) x2= 5/3, x3 = -1/3; (Non-basic) x1= 0, and (Basic) x1= 1, x3=0; (Non-basic) x2=0. In each of the two basic solutions, at least one of the basic variables is zero. Hence two of the basic solutions are degenerate solutions. The solution [o, 5/3, -1/3] is not a feasible solution. The two basic solutions [1,0,0] and [1,0,0] are basic feasible solutions. =

  42. Some Importance Theorems for Simplex Methods (without proof) Theorem1: The set of feasible solutions to an LPP is a convex set. Theorem2: [Fundamental theorem of LP] If the feasible region of an LPP is a convex polyhedron, then there exists an optimal solution to the LPP and at least one basic feasible solution must be optimal. Theorem3: [Reduction of feasible solution to a basic feasible solution] If an LPP has a feasible solution, then it also has a basic feasible solution. Theorem4: [Replacement of a Basic vector] Let an LPP have a basic feasible solution. If we drop one of the basic vectors and introduce a non-basic vector in the basic set, then the new solution obtained is also a basic feasible solution, provided these vectors are suitably selected.

  43. Sample of Simplex Table • zj = Σi=1cBi yij is called the • evaluation and the • number zj-cj is called • the net evaluation • corresponding to each column. A (Coefficient Matrix)

  44. Theorem 5: [Conditions of optimality] A sufficient condition for a basic feasible solution to an LPP to be an optimum (maximum) is that zj-cj≥0 for all j for which the column vector ajЄA is not, in the basis B. Theorem 6: [ Unbounded solution] Let there exist a basic feasible solution to a given LPP. If for at least one j, for which yij≤0 (I = 1,2,…, m), zj-cj is negative, then there does not exist any optimum solution to this LPP. Example: Solve the following LPP by Simplex Method: Max, z= x1+2x2, S.t. x1-x2≤4, x1-5x2 ≤8, x1, x2≥0 Solution: Standard form of LPP is Objective function Max, z = x1+2x2+0.x3+0.x4 Subject to the constrains: x1-x2 +x3+0.x4=4 x1-5x2 + 0.x3 + x4 =8 x1, x2, x3, x4≥0 Initial Simplex Table

  45. FLOW CHART Simplex Algorithm for Maximization Obtain an initial basic feasible solution of the problem Reformulate the given LPP as standard form Compute the net Evaluation, set up in initial Table Update the simplex table by appropriate operation (pivoting) Examine the index row of net evaluation NO Identify the Pivot/Key element. Key column and Key row meet at Pivot/key element. Is there any negative net evaluation? Optimum solution has been attained and STOP Are all elements of a column non-positives and corresponding net evaluation also negative? There exist an unbound solution to the problem, STOP YES Choose the smallest ratio. Row contains the smallest ratio is called Key row Choose the most negative net evaluation. Corresponding the most negative evaluation column is KEY column. Select the positive element from key column and divide the corresponding values of the current basic variables by them

  46. Example2: A firm manufactures 3 products A, B and C. The profits are $3, $2, and $4 respectively. The firm has 2 machines and below is the required processing time in minutes for each machine on each product. Machine G and H have 2000 and 2500 machine-minutes respectively. How many units of each product should be produced in order to gain maximum profit?

  47. Solution: Define the objective Maximize profit. Define the decision variables: x 1 = number of products of type A, x 2 = number of products of type B and x 3 = number of products of type C . Objective Function, Max Z = 3x 1 + 2x 2 + 4x 3 Subject to the constraints, 4x 1 + 3x 2 + 5x 3 ≤ 2000 2x 1 + 2x 2 + 4x 3 ≤ 2500 x 1 , x 2 , x 3 ≥ 0 The standard form of LPP is, Objective Function, Max Z = 3x1 + 2x2 + 4x3 +0.x4+0.x5 Subject to the constraints, 4x1 + 3x2 + 5x3 +x4 +0.x5 = 2000 2x1 + 2x2 + 4x3 +0.x4 +x5 = 2500 x1 , x2 , x3, x4, x5≥ 0 An initial basic feasible solution is: x1 = x2= x3 =0, then x4 = 2000 and x5 = 2500

  48. Initial Simplex Table 5 Key Row 5 is Pivot/Key element Key Column Since all elements in index row are non-positive, thus we have not attained optimum solution. The current solution can be further improved.

  49. Algorithm for Iteration Step1: Identify the KEY Column, KEY row and the leading element (pivot/key element) from previous table. Step2: Determine the departing variable: The key row and corresponding basic variable will leave the basis. Step3: Determine the introducing variable: The corresponding non-basic variable of Key column will be replaced in basic for the next table. Step 4: Compute values for the key row (previous table) for new table: To do this, simply divide every number in the key row (previous table) by the element. [ y^kj=ykj/ykr] Step 5: Compute values for each remaining row (rows) for new table: All remaining row (s) for the next simplex table can be calculated by the formula: New row number = (Number in old row) – [Corresponding number in key row] X [Corresponding fixed ratio], [y^ij = yij-(ykj/ykr)yir] where fixed ratio = old number in the key column/key element (number). Step6: Put the new values in new simplex table (First Iteration): Step7: Test for optimality and take decision after looking Index row.

More Related