ENGINEERING OPTIMIZATION Methods and Applications

1. ENGINEERING OPTIMIZATION Methods and Applications A. Ravindran, K. M. Ragsdell, G. V. Reklaitis Book Review

2. Chapter 9: Direction Generation Methods Based on Linearization Part 1: Ferhat Dikbiyik Part 2:Mohammad F. Habib Review Session July 30, 2010

3. The Linearization-based algorithms in Ch. 8 • LP solution techniques to specify the sequence of intermediate solution points. The linearized subproblem at this point is updated The exact location of next iterate is determined by LP The linearized subproblem cannot be expected to give a very good estimate of either boundaries of the feasible solution region or the contours of the objective function

4. Good Direction Search Rather than relying on the admittedly inaccurate linearization to define the precise location of a point, it is more realistic to utilize the linear approximations only to determine a locally good direction for search.

5. Outline • 9.1 Method of Feasible Directions • 9.2 Simplex Extensions for Linearly Constrained Problems • 9.3 Generalized Reduced Gradient Method • 9.4 Design Application

6. 9.1 Method of Feasible Directions G. Zoutendijk Mehtods of Feasible Directions, Elsevier, Amsterdam, 1960

7. Preliminaries Source: Dr. Muhammad Al-Slamah, Industrial Engineering, KFUPM

8. Preliminaries Source: Dr. Muhammad Al-Slamah, Industrial Engineering, KFUPM

9. 9.1 Method of Feasible Directions • Suppose that is a starting point that satisfies all constraints. • and suppose that a certain subset of these constraints are binding at .

10. 9.1 Method of Feasible Directions • Suppose is a feasible point • Define x as • The first order Taylor approximation of f(x) is given by • In order for , we have to have • A direction satisfying this relationship is called a descent direction

11. 9.1 Method of Feasible Directions • This relationship dictates the angle between d and to be greater than 90° and less than 270 °. Source: Dr. Muhammad Al-Slamah, Industrial Engineering, KFUPM

12. 9.1 Method of Feasible Directions • The first order Taylor approximation for constraints • And with assumption (because it’s binding) • In order for x to be a feasible, hence • Any direction d satisfying this relationship called a feasible direction

13. 9.1 Method of Feasible Directions • This relationship dictates the angle between d and has to be between 0° and than 90 °. Source: Dr. Muhammad Al-Slamah, Industrial Engineering, KFUPM

14. 9.1 Method of Feasible Directions • In order for x to solve the inequality constrained problem, the direction d has to be both a descent and feasible solution. Source: Dr. Muhammad Al-Slamah, Industrial Engineering, KFUPM

15. 9.1 Method of Feasible Directions • Zoutendijk’s basic idea is at each stage of iteration to determine a vector d that will be both a feasible direction and a descent direction. This is accomplished numerically by finding a normalized direction vector d and a scalar parameter θ> 0 such that and θ is as large as possible.

16. 9.1 Method of Feasible Directions Source: Dr. Muhammad Al-Slamah, Industrial Engineering, KFUPM

17. 9.1.1 Basic Algorithm • The active constraint set is defined as for some small

18. 9.1.1 Basic Algorithm • Step 1. Solve the linear programming problem Label the solution and

19. 9.1.1 Basic Algorithm • Step 2. If the iteration terminates, since no further improvement is possible. Otherwise, determine If no exists, set

20. 9.1.1 Basic Algorithm • Step 3. Find such that Set and continue.

21. Example 9.1 , since g_1 is the only binding constraint.

22. Example 9.1 We must search along the ray to find the point at which boundary of feasible region is intersected

23. Example 9.1 Since is positive for all α≥ 0 and is not violated as αis increased. To determine the point at which will be intersected, we solve Finally, we search on α over range to determine the optimum of

24. Example 9.1

25. 9.1.2 Active Constraint Sets and Jamming • Example 9.2

26. 9.1.2 Active Constraint Sets and Jamming • The active constraint set used in the basic form of feasible direction algorithm, namely, cannot only slow down the process of iterations but also lead to convergence to points that are not Kuhn-Tucker points. • This type of false convergence is known as jamming

27. 9.1.2.1 ε-Perturbation Method • At iteration point and with given , define and carry out step 1 of the basic algorithm. • Modify step 2 with the following: If , set and continue. However, if , set and proceed with line search of the basic method. If , then a Kuhn-Tucker point has been found. With this modification, it is efficient to set ε rather loosely initially so as to include the constraints in a larger neighborhood of the point . Then, as the iterations proceed, the size of the neighborhood will be reduced only when it is found to be necessary.

28. 9.1.2.2 Topkis-Veinott Variant • This approach simply dispense with the active constraint concept altogether and redefine the direction-finding subproblem as follows: If the constraint loose at , then the selection of d is less affected by constraint j, because the positive constraint value will counterbalance the effect of the gradient term. This ensures that no sudden changes are introduced in the search direction.

29. 9.2 Simplex Extensions for Linearly Constrained Problems • At a given point, the number of directions that are both descent and feasible directions is generally infinite. • In the case of linear programs, the generation of search directions was simplified by changing one variable at a time; feasibility was ensured by checking sign restrictions, and descent was ensured by selecting a variable with negative relative-cost coefficient.

30. 9.2.1 Convex Simplex Method N components M rows • Given a feasible point the x variable is partitioned into two sets: • the basic variables , which are all positive • the nonbasic variables , which are all zero an M vector an N-M vector

31. 9.2.1 Convex Simplex Method

32. 9.2.1 Convex Simplex Method The relative-cost coefficients The nonbasic variable to enter is selected by finding such that The basic variable to leave the basis is selected using the minimum-ratio rule. That is, we find r such that elements of matrix

33. 9.2.1 Convex Simplex Method • The new feasible solution and all other variables zero. At this point, the variables and are relabeled. Since an exchange will have occurred, will be redefined. The matrix is recomputed and another cycle of iterations is begun.

34. 9.2.1 Convex Simplex Method • The application of same algorithm to linearized form of a non-linear objective function: • The relative-cost factor:

35. Example 9.4

36. Example 9.4

37. Example 9.4 The relative-cost factor:

38. Example 9.4 The nonbasic variable to enter will be , since The basic variable to leave will be , since

39. Example 9.4 • The new point is thus A line search between and is now required to locate minimum of . Note that remains at 0, while changes as given by

40. Example 9.4

41. Convex Simplex Algorithm

42. Convex Simplex Algorithm

43. 9.2.2 Reduced Gradient Method • The nonbasic variable direction vector: • This definition ensures that when for all i, the Kuhn-Tucker conditions are satisfied.

44. 9.2.2 Reduced Gradient Method • In the first case, the limiting α value will be given by • If all , then set • In the second case, If all , then set

45. Reduced Gradient Algorithm