Optimization Methods: Gradient Descent and Conjugate Search

1. General Strategy for Gradient methods (1) Calculate a search direction (2) Select a step length in that direction to reduce f(x) Chapter 6 Steepest Descent Search Direction Don’t need to normalize Method terminates at any stationary point. Why?

2. Chapter 6

3. Chapter 6

4. Numerical Method Use coarse search first (1) Fixed a (a = 1) or variable a (a = 1, 2, ½, etc.) Options for optimizing a (1) Use interpolation such as quadratic, cubic (2) Region Elimination (Golden Search) (3) Newton, Secant, Quasi-Newton (4) Random (5) Analytical optimization (1), (3), and (5) are preferred. However, it may not be desirable to exactly optimize a (better to generate new search directions). Chapter 6

5. Chapter 6

6. Chapter 6

7. Conjugate Search Directions • Improvement over gradient method for general quadratic functions • Basis for many NLP techniques • Two search directions are conjugate relative to Q if • To minimize f(xnx1) when H is a constant matrix (=Q), you are guaranteed to reach the optimum in n conjugate direction stages if you minimize exactly at each stage (one-dimensional search) Chapter 6

8. Chapter 6

9. Conjugate Gradient Method by minimizing f(x) with respect to a in the s0 direction (i.e., carry out a unidimensional search for a0). Step 3.Calculate The new search direction is a linear combination of Chapter 6 For the kth iteration the relation is (6.6) For a quadratic function it can be shown that these successive search directions are conjugate. After n iterations (k = n), the quadratic function is minimized. For a nonquadratic function, the procedure cycles again with xn+1 becoming x0. Step 4. Test for convergence to the minimum of f(x). If convergence is not attained, return to step 3. Step n. Terminate the algorithm when is less than some prescribed tolerance.