An Introduction to Optimization Theory

1. An Introduction to Optimization Theory

2. Outline • Introduction • Unconstrained optimization problem • Constrained optimization problem

3. Introduction • Mathematically speaking, optimization is the minimization of a objective function subject to constraints on its variables. Mathematically, we have

4. Introduction

5. Introduction-Linear regression

6. Introduction-Battery charger

7. Unconstrained optimization problem • Definition for unconstrained optimization problem:

8. Unconstrained optimization problem

9. Gradient descent algorithm

10. Gradient descent algorithm • Gradient descent algorithm may be trapped into the local extreme instead of the global extreme

11. Gradient descent algorithm • Methodology for choosing suitable step size αk ---- Steepest descent algorithm

12. Gradient descent algorithm

13. Gradient descent algorithm • Steepest descent algorithm with quadratic cost function:

14. Gradient descent algorithm Update equation:

15. Newton method • Summary for Newton method

16. Newton method

17. Newton method • Procedure for Newton method

18. Quasi-Newton method

19. Quasi-Newton method • What properties of F(x(k))-1 should it mimic ? 1. Hk should be a symmetric matrix 2. Hk should with secant property

20. Quasi-Newton method • Typical approaches for Quasi-Newton method 1. Rank-one formula 2. DFP algorithm 3. BFGS algorithm (L-BFGS , L indicates limited-memory)

21. Constrained optimization problem • Definition for constrained optimization problem

22. Problems with equality constraints ---- Lagrange multiplier

23. Problems with equality constraints ---- Lagrange multiplier

24. Problems with equality constraints ---- Lagrange multiplier

25. Problems with equality constraints ---- Lagrange multiplier • Suppose x* is a local minimizer

27. Karush-Kuhn-Tucker condition (KKT) • From now on, we will consider the following problem

28. Karush-Kuhn-Tucker condition (KKT) Note that:

29. Projection Constrained set Ω Initial solution Image statistics & Image enhancement • Illustration for gradient descent with projection

30. Useful Matlab introductions for optimization • Useful instructions included in Matlab for optimization 1. fminunc: Solver for unconstrained optimization problems 2. fmincon: Solver for constrained optimization problems 3. linprog: Solver for linear programming problems 4. quadprog: Solver for quadratic programming problems