15.053 Thursday, April 25

1. 15.053 Thursday, April 25 • Nonlinear Programming Theory • Separable programming Handouts: Lecture Notes

2. Difficulties of NLP Models Linear Program: Nonlinear Programs:

3. Graphical Analysis of Non-linear programs in two dimensions: An example • Minimize • subject to (x - 8)2 + (y - 9)2≤ 49 x ≥ 2 x ≤ 13 x + y ≤ 24

4. Where is the optimal solution? Note: the optimal solution is not at a corner point. It is where the isocontour first hits the feasible region.

5. Another example: Minimize (x-8)2 + (y-8)2 Then the global unconstrained minimum is also feasible. The optimal solution is not on the boundary of the feasible region.

6. Local vs. Global Optima Def’n: Let x be a feasible solution, then – x is a global maxif f(x) ≥ f(y) for every feasible y. – x is a local maxif f(x) ≥ f(y) for every feasible y sufficiently close to x (i.e., xj-ε ≤ yj ≤ xj+ ε for all j and some small ε). There may be several locally optimal solutions.

7. When is a locally optimal solution also globally optimal? • We are minimizing. The objective • function is convex. The feasible • region is convex.

8. Convexity and Extreme Points We say that a set S is convex, if for every two points x and y in S, and for every real number λ in [0,1], λx + (1-λ)y ε S. The feasible region of a linear program is convex. We say that an element w ε S is an extreme point (vertex, corner point), if w is not the midpoint of any line segment contained in S.

9. Recognizing convex feasible regions • If all constraints are linear, then the • feasible region is convex • The intersection of convex regions is • convex • If for all feasible x and y, the • midpoint of x and y is feasible, then • the region is convex (except in • totally non-realistic examples. )

10. Which are convex?

11. Convex Functions • Convex Functions: f(ly + (1- l)z) £ lf(y) + (1- l)f(z) • for every y and z and for 0£ l £1. e.g., f((y+z)/2) £f(y)/2 + f(z)/2 • We say “strict” convexity if sign • is “<” for 0< λ <1. Line joining any points is above the curve

12. Convex Functions • Convex Functions: f(ly + (1- l)z) ≥ lf(y) + (1- l)f(z) • for every y and z and for 0£ l £1. e.g., f((y+z)/2) ≥f(y)/2 + f(z)/2 • We say “strict” convexity if sign • is “<” for 0< λ <1. Line joining any points is above the curve

13. Classify as convex or concave or both or neither.

14. What functions are convex? • f(x) = 4x + 7 all linear functions • f(x) = 4x2 – 13 some quadratic functions • f(x) = ex • f(x) = 1/x for x > 0 • f(x) = |x| • f(x) = - ln(x) for x > 0 • Sufficient condition: f”(x) > 0 for all x.

15. What functions are convex? • If f(x) is convex, and g(x) is convex. • Then so is • h(x) = a f(x) + b g(x) for a>0, b>0. • If y = f(x) is convex, then • {(x,y) : f(x) ≤ y} is a convex set

16. Local Maximum (Minimum) Property • A local max of a concave function on a convex • feasible region is also a global max. • A local min of a convex function on a convex feasible • region is also a global min. • Strict convexity or concavity implies that the global • optimum is unique. • Given this, we can exactly solve: • – Maximization Problems with a concave objective • function and linear constraints • – Minimization Problems with a convex objective • function and linear constraints

17. Which are convex feasible regions? (x, y) : y ≤ x2 + 2 (x, y) : y ≥ x2 + 2 (x,y) : y = x2 + 2

18. More on local optimality • The techniques for non-linear optimization • minimization usually find local optima. • This is useful when a locally optimal • solution is a globally optimal solution • It is not so useful in many situations. • Conclusion: if you solve an NLP, try to • find out how good the local optimal • solutions are.

19. Solving NLP’s by Excel Solver

20. Finding a local optimal for a single variable NLP Solving NLP's with One Variable: max f(θ) s.t. a ≤ θ ≤ b Optimal solution is either a boundary point or satisfies f′(θ∗) = 0 and f ″(θ∗) < 0.

21. Solving Single Variable NLP (contd.) If f(θ) is concave (or simply unimodal) and differentiable max f(θ) s.t. a ≤ θ ≤ b Bisection (or Bolzano) Search: • Step 1. Begin with the region of uncertainty for θ as [a, • b]. Evaluate f ′ (θ) at the midpoint θΜ =(a+b)/2. • Step 2. If f ′ (θΜ) > 0, then eliminate the interval up to θΜ. • If f′(θΜ) < 0, then eliminate the interval beyond θΜ. • Step 3. Evaluate f′(θ) at the midpoint of the new • interval. Return to Step 2 until the interval of uncertainty • is sufficiently small.

22. Unimodal Functions • A single variable function f is • unimodalif there is at most one local • maximum (or at most one local minimum) .

23. Other Search Techniques • Instead of taking derivatives (which may be • computationally intensive), use two function • evaluations to determine updated interval. • Fibonacci Search • Step 1. Begin with the region of uncertainty for θ as [a, • b]. Evaluate f (θ1) and f (θ2) for 2 symmetric points • θ1<θ2. • Step 2. If f (θ1) ≤ f (θ2), then eliminate the interval up tθ1. • If f (θ1) > f (θ2), then eliminate the interval beyond θ2. • Step 3. Select a second point symmetric to the point • already in the new interval, rename these points θ1 and • θ2 such that θ1<θ2 and evaluate f (θ1) and f (θ2). Return • to Step 2 until the interval is sufficiently small.

24. On Fibonacci search 1, 1, 2, 3, 5, 8, 13, 21, 34 At iteration 1, the length of the search interval is the kth fibonacci number for some k At iteration j, the length of the search interval is the k-j+1 fibonacci number. The technique converges to the optimal when the function is unimodal.

25. Finding a local maximum using Fibonacci Search. Length of search interval Where the maximum may be

26. The search finds a local maximum, but not necessarily a global maximum.

27. The search finds a local maximum, but not necessarily a global maximum.

28. Number of function evaluations in Fibonacci Search • As new point is chosen symmetrically, the length lk of • successive search intervals is given by: lk= lk+1 + lk+2 . • Solving for these lengths given a final interval length of • 1, ln = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, • 34,… • Thus, if the initial interval has length 34, it takes 8 • function calculations to reduce the interval length to 1. • Remark: if the function is convex or unimodal, then • fibonacci search converges to the global maximum

29. Separable Programming • Separable programs have the form: Max st Each variable xj appears separately, one in each function gij and one in each function fj in the objective. Each non-linear function is of a single variable.

30. Examples of Separable Programs f(x1,x2) = x1(30-x1)+x2(35-x2)-2x12-3x22 3 f(x1,x2) = x15 + -18e-x2-4x2 x1 f(x1,x2,x2) = lnx15-sinx2-x3e-x3)+7x1-4

31. Approximating a non-linear function with a piecewise linear function. • Aspect 1. Choosing the approximation. • Aspect 2. When is the piecewise linear • approximation a linear program in • disguise?

32. Approximating a non-linear function of 1 variable

33. Approximating a non-linear function of 1 variable: the λmethod Choose different values of x to approximate the x-axis Approximate using piecewise linear segments

34. More on the λmethod a1 = -3, f(a1) = -20 a2 = -1 f(a2) = -7 1/3 Suppose that for –3 ≤ x ≤ -1, we represent x hasλ1 (-3) +λ2 (-1) whereλ1 +λ2 = 1 andλ1,λ2≥ 0 Then we approximate f(x) as λ1 (-20) +λ2 (-7 1/3)

35. More on the λmethod Suppose that for –1 ≤ x ≤ 1, we represent x hasλ2 (-3) +λ3 (-1) whereλ2 +λ3 = 1 andλ2,λ3≥ 0 a2 = -1, f(a2) = -7 1 /3 a3 = 1 f(a3) = -2 2/3 How do we approximate f( ) in this interval? What if –3 ≤ x ≤ 1?

36. Almost the λ method Original problem: min x3/3 + 2x – 5 + more terms s.t. -3 ≤ x ≤ 3 + many more constraints a1 = -3; a2 = - 1; a3 = 1; a4 = 3 f(a1) = -20; f(a2) = -7 1/3; f(a3) = -2 2/3; f(a4) = 4 Approximated problem: minλ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4) more linear terms s.t. λ1 + λ2 + λ3 + λ4= 1 ; λ ≥ 0 + many more constraints

37. Why the approximation is incorrect Approximated problem: minλ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4) more linear terms s.t. λ1+λ2 +λ3+λ4 = 1 ; λ ≥ 0 Considerλ1= 1/2 ;λ2=0 ;λ3=1/2 ;λ4 = 0; The method gives the correct approximation if only two consecutiveλs are positive.

38. Adjacency Condition 1. At most two weights (λs) are positive 2. If exactly two weights (λs) are positive, then they are λj and λj+1 for some j 3. The same condition applies to every approximated function.

39. Approximating a non-linear objective • function for a minimization NLP. • original problem: minimize {f(y): y ∈ P} • Suppose that y = Σjλjaj , • where Σjλj = 1 and λ >= 0 . • Approximate f(y). • minimize {Σjλjf(aj): Σjλjaj∈ P} • Note: when given a choice of representing y in • alternative ways, the LP will choose one that • leads to the least objective value for the • approximation.

40. For minimizing a convex function, the λ-method automatically satisfies the additional adjacency property. min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; λ ≥ 0 + adjacency condition + other constraints

41. Feasible approximated objective functions without the adjacency conditions min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; λ ≥ 0 + other constraints

42. But a minimum in this case always occurs on the piecewise linear curve. min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; λ ≥ 0 + other constraints

43. Separable Programming (in the case of linear constraints) • Max f ( )x • s.t. Dx =d • x ³ 0 • Begin with an NLP: • Transform to Separable: • Approximate using the λ Method: n • Max∑ fj (xj) =s.t. Dx =d x ³ 0 j=1

44. Approximation • Re-express in terms of λ variables: and the adjacency conditions If the original problem is concave, then you can drop the adjacency conditions (they are automatically satisfied.)

45. How can one construct separable functions? Term Substitution Constraints Restriction None None None

46. Transformation Examples Ex : (x1 + x2 + x3)6 Substitute y6 and let y = x1 + x2 + x3 x1x22 let y1 Ex : and let y2 1+ x3 and add the constraint

47. NLP Summary • Convex and Concave functions as well as convex sets • are important properties • Bolzano and Fibonacci search techniques • – used to solve single variable unimodal functions • Separable Programming • – nonlinear objective and nonlinear constraints that • are separable • – General approximation technique