Linear & Nonlinear Programming -- Basic Properties of Solutions and Algorithms

1. Linear & Nonlinear Programming --Basic Properties of Solutions and Algorithms

2. Outline • First-order Necessary Condition • Examples of Unconstrained Problems • Second-order Conditions • Convex and Concave Functions • Minimization and Maximization of Convex Functions • Global Convergence of Descent Algorithms • Speed of Convergence

3. Introduction • Considering optimization problem of the form where f is a real-valued function and Ω, the feasible set, is a subset of En.

4. Weierstras Theorem • If f is continuous and Ω is compact, a solution point of minimization problem exists.

5. Two Kinds of Solution Points • Definition of a relative minimum point • A point is said to be a relative minimum point of f over Ω if there is an such that f(x) ≥ f(x*) for all within a distance of x*. If f(x) >f(x*) for all , x≠x*, within a distance of x*,then x* is said to be a strict relative minimum point of f over . • Definition of a global minimum point • A point is said to be a global minimum point of f over Ω if f(x) ≥ f(x*) for all . If f(x) >f(x*) for all , x≠x*, then x* is said to be a strict global minimum point of f over .

6. Two Kinds of Solution Points (cont’d) • We can achieve relative minimum by using differential calculus or a convergent stepwise procedure. • Global conditions and global solutions can, as a rule, only be found if the problem possesses certain convexity properties that essentially guarantee that any relative minimum is a global minimum.

7. Feasible Directions • To derive necessary conditions satisfied by a relative minimum point x*, the basic idea is to consider movement away from the point in some given direction. • A vector d is a feasible direction at x if there is an such that for all α, .

8. Feasible Directions (cont’d) • Proposition 1 ( first-order necessary conditions) • Let Ω be a subset of En and let be a function on Ω. If x* is a relative minimum point of f over Ω, then for any that is a feasible direction at x*, we have Proof :

9. Feasible Directions (cont’d) • Corollary ( unconstrained case ) • Let Ω be a subset of En and let be a function on Ω. If x* is a relative minimum point of f over Ω and x* is an interior point of Ω, then . Since in this case d can be any direction from x*, and hence for all . This implies .

10. Example 1 about Feasible Directions • Example 1 ( unconstrained ) minimize There are no constrains, so Ω = En These have the unique solution x1=1, x2=2, so it is a global minimum of f .

11. Example 2 about Feasible Directions • Example 2 (a constrained case) minimize subject to x1≥ 0, x2 ≥ 0. since we know that there is a global minimum at x1=1/2, x2=0, then

12. Example 2 (cont’d) x2 3/2 1 Feasible region 1/2 d = (d1 , d2) 0 x1 1 (1/2, 0) 3/2 2

13. Example 3 of Unconstrained Problems • The problem faced by an electric utility when selecting its power-generating facilities. Its power-generating requirements are summarized by a curve, h(x), as shown in Fig.6.2(a), which shows the total hours in a year that a power level of at least x is required for each x. For convenience the curve is normalized so that the upper limit is unity. The power company may meet these requirements by ins-talling generating equipment, such as (1) nuclear or (2) coal-fired, or by purchasing power from a central energy.

14. Example 3 (cont’d) • Associated with type i ( i = 1,2 ) of generating equipment is a yearly unit capital cost bi and a unit operating cost ci. The unit price of power purchased from the grid is c3. The requirements are satisfied as shown in Fig.6.2(b), where x1 and x2 denote the capacities of the nuclear and coal-fired plants, respectively.

15. hours required h(x) 1 power (megawatts) (a) Example 3 (cont’d) hours required h(x) coal nuclear purchase x1 x2 1 power (megawatts) (b) Fig.6.2 Power requirement curve

16. Example 3 (cont’d) • The total cost is • And the company wishes to minimize the set defined by x1 ≥ 0 , x2 ≥ 0 , x1+x2 ≤ 1

17. Example 3 (cont’d) • Assume that the solution is interior to the constraints, by setting the partial derivatives equal to zero, we obtain the two equations which represent the necessary conditions • In addition, If x1=0, then equality (1) relax to ≥ 0 If x2=0, then equality (2) relax to ≥ 0

18. Second-order Conditions • Proposition 1 ( second-order necessary conditions ) • Let Ω be a subset of En and let be a function on Ω. if x* is a relative minimum point of f over Ω, then for any which is a feasible direction at x* we have i) ii) if , then Proof : The first condition is just propotion1, and the second condition applies only if .

19. Proposition 1 (cont’d) • Proof (cont’d) :

20. Example 1 about Proposition 1 • For the same problem as Example 2 of Section 6.1, we have d = (d1, d2) • Thus condition (ii) of Proposition 1 applies only if d2 = 0. In that case we have so condition (ii) is satisfied.

21. Proposition 2 • Proposition 2 (unconstrained case) • Let x* be an interior point of the set Ω, and suppose x* is a relative minimum point over Ω of the function . Then i) ii) for all d, . • It means that F(x*), simplified notation of , is positive semi-definite.

22. Example 2 about Proposition 2 • Consider the problem • If we assume the solution is in the interior of the feasible set, that is, if x1 > 0, x2 > 0, then the first-order necessary conditions are

23. Example 2 (cont’d) • Boundary solution is x1 = x2 = 0 • Another solution atx1 = 6, x2 = 9 • If we fixed x1 at x1 = 6, then the relative minimum with respect to x2 at x2 = 9. • Conversely, with x2 fixed at x2 = 9, the objective attains a relative minimum w.r.t.x1at x1 = 6. • Despite this fact, the point x1 = 6, x2 = 9 is not a relative minimum point, because the Hessian matrix F(x*)x1 = 6, x2 = 9 is not a positive semi-definite since its determinant is negative.

24. Sufficient Conditions for a Relative Minimum • We give here the conditions that apply only to unconstrainedproblems, or to problems where the minimum point is interior to the feasible solution. • Since the corresponding conditions for problems where the minimum is achieved on a boundary point of the feasible set are a good deal more difficult and of marginal practical or theoretical value. • A more general result, applicable to problems with functional constrains, is given in Chapter10.

25. Proposition 3 • Proposition 3 (2-order sufficient conditions-unconstrained case) • Let be a function defined on a region in which the point x* is an interior point. Suppose in addition that i) • is positive definite Then x* is a strict relative minimum point of f. Proof : since is positive definite , there is an a > 0 such that for all d, . Thus by Taylor’s Theorem

26. Convex Functions • Definition • A function f defined on a convex set Ω is said to be convex. If, for every and every α, 0 ≤ α ≤ 1, there holds If, for every α, 0 <α < 1, and x1≠x2, there holds then f is said to be strictly convex.

27. Concave Functions • Definition • A function g defined on a convex set Ω is said to be concave if the function f = -g is convex. The function g is strictly concave if -g is strict convex.

28. f x1 x2 αx1+(1-α)x2 Graphs of Strict Convex Function

29. Graphs of Convex Function f x1 x2 αx1+(1-α)x2

30. f x1 x2 αx1+(1-α)x2 Graphs of Concave Function

31. f x1 x2 Graph of Neither Convex or Concave

32. Combinations of Convex Functions • Proposition 1 • Let f1 and f2 be convex function on the convex set Ω. Then the f1+f2 is convex on Ω. • Proposition 2 • Let f be a convex function over the convex set Ω. Then af is convex for any a ≥ 0.

33. Combinations (cont’d) • Through the above two propositions it follows that a positive combination a1 f1+a1 f2+…am fm of is again convex.

34. Convex Inequality Constrains • Proposition 3 • Let f be a convex function on a convex set Ω. The set is a convex for every real number c.

35. Proof • Let , then and for 0 <α< 1 , Thus

36. Properties of Differentiable Convex Functions • Proposition 4 • Let , then f is convex over a convex set Ω if and only if for all

37. Recall • The original definition essentially states that linear interpolation between two points overestimates the function, • while here stating that linear approximation based on the local derivative underestimates the function.

38. f x1 x2 αx1+(1-α)x2 Recall (cont’d) f is a convex function between two points

39. Recall (cont’d) f(y) y x

40. Two Continuously Differentiable Functions • Proposition 5 • Let , then f is convex over a convex set Ω containing an interior point if only if the Hessian matrix F of f is positive semi-definite through Ω.

41. Proof • By Taylor’s theorem we have for some α, 0 ≤α≤ 1. if the Hessian is everywhere positive semi-definite, we have

42. Minimization and Maximization of Convex Functions • Theorem 1 • Let f be a convex function defined on the convex set Ω, then the set where f achieves its minimum is convex, and any relative minimum of f is a global minimum. • Proof (contradiction)

43. Minimization and Maximization of Convex Functions (cont’d) • Theorem 2 • Let be a convex on the convex set Ω. If there is a point such that, for all then x* is a global minimum point of f over Ω. • Proof

44. Minimization and Maximization of Convex Functions (cont’d) • Theorem 3 • Let f be a convex function defined on the bounded, closed convex set Ω. If fhas a maximumover Ω, then it is achieved at an extreme point of Ω.

45. Global Convergence of Descent Algorithms • A good portion of the remainder of this book is devoted to presentation and analysis of various algorithms designed to solve nonlinear programming problems. However, they have the common heritage of all being iterative descent algorithms. • Iterative • The algorithm generated a series of points, each point being calculated on the basis of the points preceding it. • Descent • As each new point is generated by the algorithm the corresponding value of some function (evaluated at the most recent point) decreases in values.

46. Global Convergence of Descent Algorithms (cont’d) • Globally convergent • If for arbitrary starting points the algorithm is guaranteed to generate a sequence of points converging to a solution, then the algorithm is said to be globally convergent.

47. Algorithm and Algorithmic Map • We formally define an algorithm A as a mapping taking points in a space Xinto other points in X, then the generated sequence { xk } defined by • With this intuitive idea of an algorithm in mind, we now generalize the concept somewhat so as to provide greater flexibility in our analysis. • Definition • An algorithm A is a mapping defined on a space X that assigns to every point a subset of X.

48. Mappings • Given the algorithm yields A(xk ) which is a subset of X. From this subset an arbitrary elementxk+1 is selected. In this way, given an initial point x0, the algorithm generates sequences through the iteration • The most important aspect of the definition is that the mapping A is apoint-to-set mapping of X.

49. Example 1 • Suppose for x on the real line we define so that A(x) is an interval of the real line. Starting at x0 = 100, each of the sequences below might be generated from iterative application of this algorithm. 100, 50, 25, 12, -6, -2, 1, 1/2,… 100, -40, 20, -5, -2, 1, 1/4, 1/8,… 100, 10, 1/16, 1/100, -1/1000,…

50. Descent • Definition • Let be a given solution set and let A be an algorithm on X. A continuous real-valued functions Z on X is said to be a descent function for and A if it satisfies