Chapter 5

1. Chapter 5 One Dimensional Search Chapter 5

2. Unidimensional Search • If have a search direction, want to minimize in that • direction by numerical methods Chapter 5 • Search Methods in General • 2.1. Non Sequential – Simultaneous evaluation of f at n points – no good (unless on parallel computer). • 2.2. Sequential – One evaluation follows the other.

3. Types of search that are better or best is often • problem dependent. Some of the types are: • a. Newton, Quasi-Newton, and Secant methods. • b. Region Elimination Methods (Fibonacci, Golden • Section, etc.). • c. Polynomial Approximation (Quadratic Interpolation, • etc.). • d. Random Search • Most methods assume • (a) a unimodal function, (b) that the min is • bracketed at the start and (c) also you start in a • direction that reduces f. Chapter 5

4. To Bracket the Minimum Chapter 5

5. Chapter 5

6. 1. Newton’s Method Newton’s method for an equation is Chapter 5 Application to Minimization The necessary condition for f(x) to have a local minimum is f′(x) = 0. Apply Newton’s method.

7. Examples Minimize Chapter 5 Minimize

8. Advantages of Newton’s Method • (1) Locally quadratically convergent (as long as f′(x) is • positive – for a minimum). • For a quadratic function, get min in one step. • Disadvantages • Need to calculate both f′(x) and f″(x) • If f″(x)→0, method converges slowly • If function has multiple extrema, may not converge • to global optimum. Chapter 5

9. 2. Finite-Difference Newton Method Replace derivatives with finite differences Chapter 5 Disadvantage Now need additional function evals (3 here vs. 2 for Newton)

10. 3. Secant(Quasi-Newton) Method Analogous equation to (A) is The secant approximates f″(x) as a straight line Chapter 5

11. Start the Secant method by using 2 points spanning x at which first derivatives are of opposite sign. For next stage, retain either x(q) or x(p) so that the pair of derivatives still have opposite sign. Chapter 5

12. Order of Convergence Can be expressed in various ways. Want to consider how Chapter 5 usually slow in practice

13. Fastest in practice If p = 2, quadratic convergence p = 1.32 ? Chapter 5 Usually fast in practice Some methods can show theoretically what the order is.

14. Chapter 5

15. Chapter 5

16. Chapter 5

17. Chapter 5

18. Chapter 5

19. Quadratic Interpolation Approximate f(x) by a quadratic function. Use 3 points Chapter 5

20. Chapter 5 (or use Gaussian elimination)

21. Chapter 5

22. Chapter 5

23. Chapter 5

24. Chapter 5

25. Chapter 5

26. Chapter 5

27. Chapter 5

28. Chapter 5