L11 Optimal Design L.Multipliers

1. L11 Optimal Design L.Multipliers • Homework • Review • Convex sets, functions • Convex Programming Problem • Summary

2. Constrained OptimizationLaGrange Multiplier Method • Remember: • Standard form • Max problems f(x) = - F(x)

3. KKT Necessary Conditions for Min Regularity check - gradients of active inequality constraints are linearly independent

4. Prob 4.120

5. KKT Necessary Conditions

6. Case 1

7. Case 2 Regularity: 1. pt is feasible 2. only one active constraint Point is KKT pt, OK!

8. Case 3

9. Case 4

10. Sensitivity or Case 2 Constraint Variation Sensitivity From convexity theorems: 1. Constraints linear 2. Hf is PD Therefore KKT Pt is global Min!

11. Graphical Solution 3 4 1 2

12. LaGrange Multiplier Method • May produce a KKT point • A KKT point is a CANDIDATE minimum It may not be a local Min • If a point fails KKT conditions, we cannot guarantee anything…. The point may still be a minimum. • We need a SUFFICIENT condition

13. Recall Unconstrained MVO For x* to be a local minimum: 1rst order Necessary Condition 2nd order Sufficient Condition i.e. H(x*) must be positive definite

14. Considerationsfor Constrained Min? • Objective function • Differentiable, continuous i.e. smooth? • Hf(x) Positive definite • (i.e. convexity of f(x) ) • Weierstrass theorem hints: • x closed and bounded • x contiguous or separated, pockets of points? • Constraints h(x) & g(x) • Define the constraint set, i.e. feasible region • x contiguous or separated, pockets of points? • Convex: sets, functions, constraint set and • Programming problem

15. Punchline (Theorem 4.10, pg 165) • The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if: • 1. f(x) is convex • Hf(x) Positive definite • 2. x is defined as a convex feasible set. • Equality constraints must be linear • Inequality constraints must be convex • HINT: linear functions are convex!

16. Convex sets Convex set: All pts in feasible region on a straight line(s). Non-convex set Pts on line are not in feasible region

17. Single variable No “gaps” in feasible “region”

18. Multiple variables Fig 4.21 What if it were an equality constraint? misprint

19. Figure 4.22 Convex function f(x)=x2 Bowl that holds water. .

20. Fig 4.23 Characterization of a convex function.

21. Test for Convex Function Difficult to use above definition! However, Thm 4.8 pg 163: If the Hessian matrix of the function is PD ro PSD at all points in the set S, then it is convex. PD… “strictly” convex, otherwise PSD… “convex”

22. Theorem 4.9 Given: S is convex if: 1. hi are linear 2. gj are convex i.e. Hg PD or PSD When f(x) and S are convex= “convex programming problem”

23. “Sufficient” Theorem 4.10, pg 165 • The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if: • 1. f(x) is convex • Hf(x) Positive definite • 2. x is defined as a convex feasible set S • Equality constraints must be linear • Inequality constraints must be convex • HINT: linear functions are convex!

24. Summary • LaGrange multipliers are the instantaneous rate of change in f(x) w.r.t. relaxing a constraint. • KKT point is a CANDIDATE min! (need sufficient conditions for proof) • Convex sets assure contiguity and or the smoothness of f(x) • KKT pt of a convex progamming problem is a GLOBAL MINIMUM!