Chap 3. Optimum Design Concepts

1. Global Minimum : A function f(x) of n variables has global minimum at if for all x in the set S (feasible region) Local Minimum : A function f(x) of n variable has local minimum at if for all x in a small neighborhood N of in the set S Chap 3. Optimum Design Concepts Intelligent System Design Lab.

2. Existence of global minimum If f(x) is continuous on the feasible set S which is closed and bounded, then f(x) has a global minimum in S A set S is closed if it contains all its boundary pts. A set S is bounded if it is contained in the interior of some circle compact = closed and bounded Weierstrass Theorem Intelligent System Design Lab.

3. EX : not closed not bounded If Weierstrass theorem is satisfied, existence of global optimum is guaranteed. If not, global optimum may still exist. Intelligent System Design Lab.

4. Gradient Vector : Gradient Vector tangent to the curve maximum increase in the function Normal to the surface HW : Show gradient vector is the normal to the surface Intelligent System Design Lab.

5. A matrix of second partial derivatives for the function Hessian Matrix Intelligent System Design Lab.

6. Taylor Series Expansion Intelligent System Design Lab.

7. Intelligent System Design Lab.

8. Quadratic Form A is symmetric Intelligent System Design Lab.

9. Intelligent System Design Lab.

10. Differentiation of a Quadratic Form Positive definite, positive semi definite , negative definite, negative semi definite, indefinite is a positive definite iff all A is positive definite iff all principal minors are positive. Quadratic form – Hessian(matrix) – convex function – global optimum Intelligent System Design Lab.

11. Necessary condition : Condition that must be satisfied at optimum point. Non-optimum point can still satisfy necessary condition. If a point satisfies sufficient condition, it is optimum. Necessary and Sufficient Conditions Intelligent System Design Lab.

12. Unconstrained optimization problem is not practical. However, constrained optimizations are logical extension of unconstrained Let be a local minimum point for Unconstrained Optimization Problems Intelligent System Design Lab.

13. First-order Necessary Condition The only way to make regardless of is : first-order necessary condition for local minimum stationary points : local minimum, maximum, saddle points Optimality Condition Intelligent System Design Lab.

14. Second-order Necessary Condition In general case, By adding or multiplying a const, the shape of the curve is not changed. Only value of function changed. Sufficient Condition sufficient condition Intelligent System Design Lab.

15. Necessary conditions : Equality Constraints A point satisfying is said to be a regular point of design space if are lineary independent. Lagrange Multiplier Constrained Optimization Problems Intelligent System Design Lab.

16. At the candidate minimum point, gradients of the cost and constraint func are along the same line.(In other words, is a linear combination of Lagrange function is stationary with respect to both and Therefore constrained optimization is converted to unconstrained optimization. geometrical meaning Intelligent System Design Lab.

17. Necessary Conditions : Inequality Constraints Intelligent System Design Lab.

18. Intelligent System Design Lab.

19. Let be a regular point of the constraint set that is a local minimum for subject to the constraints Define the Lagrange function for the problem as Then there exist Lagrange multipliers (a p-vector) and (an m-vector) such that the Lagrangian is stationary with respect to and i.e. Kuhn-Tucker(K-T) Necessary Condition Intelligent System Design Lab.

20. Where all derivatives are evaluated at point linear combination switching conditions complementary slackness condition active constraint feasible In general, m inequality constraints generate 2m switching conditions Intelligent System Design Lab.

21. 1) If f(x) is continuous on a closed and bounded feasible region, by Wierstrass Theorem, there exists a global minimum, choose the smallest from local minimum 2) Any local minimum is also a global minimum if OP is convex. Convex Sets A subset is said to be a convex set if every combination of any pair of pts in S i.e., if , then for all Global Optimality Intelligent System Design Lab.

22. A func f(x) is called convex on the convex set S iff where A func defined on a convex set S is convex iff is pos semi def or pos def. If is pos def, then f is called a strictly convex func. Convex Functions Intelligent System Design Lab.

23. Convex Programming Problem If func for i=1 to m are convex, then the set for i=1 to m is also convex Let a set S, S is convex if are convex and are linear. ⇒ S is nonconvex if is nonlinear If a cost func is convex over convex set S, then this OP is called convex programming problem. Thrm : If is a local min for a convex func defined on a convex set S, then it is also a global min. Transformation of a constraint may affect its convexity. However, convexity of the feasible region is not affected. Intelligent System Design Lab.

24. Summary • A function is convex if and only if its Hessian is at least positive semidefinite or positive definite at all points in the domain of the function; it is called strictly convex if the Hessian is positive definite at all points. • A linear equality or inequality constraint always defines a convex feasible region for the problem. • A nonlinear equality constraint always defines a nonconvex feasible region for the problem. • If all the equality constraint functions are linear and all the inequality constraint functions of a problem written in the standard form (minimization of a function with equality and “ type” inequality constraints) are convex, the feasible region is convex ; otherwise it may or may not be convex. Intelligent System Design Lab.

25. 5. If the cost function is convex over the convex feasible region, the problem is called the convex programming problem. 6. For a convex programming problem, Kuhn-Tucker first-order necessary conditions are also sufficient, and any local minimum is also a global minimum. 7. Nonconvex problems can also have global minimum points. Intelligent System Design Lab.

26. Second-order NC and SC for constrained Optimization SC for convex problems For convex programming problems, the first-order Kuhn-Tucker NC becomes SC. This soln will be a global min. Second-order NC for constrained problems Let satisfy the first-order K-T NC SC for constrained problems Let satisfy the first-order K-T NC. Intelligent System Design Lab.

27. Strong SC Let satisfy the first-order K-T NC. Post Optimality Analysis = Physical meaning of Lagrange Multiplies If constraints are changed Intelligent System Design Lab.

28. Constraint Variation Sensitivity Theorem Let f(x), hi(x), I=1 to p, and gi(x), j=1 to m, have two continuous derivatives. Let x* be a regular point that, together with the multipliers uj* and vi*, satisfies both the Kuhn-Tucker necessary conditions and the sufficient conditions of Theorem 3.12 for an isolated local minimum point for the problem defined in Eqs(3.37) to (3.39). If for each gj(x*)=0, it is true that uj*>0, then the solution x*(b,e) of the above modified optimization problem defined in Eqs(3.69) and (3.70) is continuously differentiable function of b and e in some neighborhood of b=0 and e=0. Intelligent System Design Lab.

29. Furthermore, Intelligent System Design Lab.

30. Ex. Intelligent System Design Lab.

31. Effect of scaling cost function The multiplies with relatively larger values will have a significant effect on optimum cost if constraints are changed Intelligent System Design Lab.

32. Effect of scaling a constraint Intelligent System Design Lab.