Method of Hooke and Jeeves

1. Method of Hooke and Jeeves

2. Multidimensional Search Without Using Derivatives In this section we consider the problem of minimizing a function f of several variables without using derivatives. The methods described here proceed in the following manner.

3. Multidimensional Search Without Using Derivatives Given a vector x, a suitable direction d is first determined, and then f is minimized from xin the direction d by one of the techniques discussed earlier. X = ( x1 , x2 , … , xn) Direction = dx Minimizing F(x) in dxdirection

4. What is the problem? We are required to solve a line search problemof the form to: minimize f(x + λd) subject to: λ L where L is typically of the form: L = {λ : λ ≥ 0 } or L = R L = { λ : a ≤ λ ≤ b }

5. What is the problem? In the statements of the algorithms, for the purpose of simplicity we have assumed that a minimizing point exists. However, this may not be the case. Here, the optimal objective value of the line search problem may be: • Unbounded: Then the original problem is unbounded and we may stop. • Finite but not achieved at any particular λ: Then λ could be chosen as such that f(x+ d) is sufficiently close to the value: inf { f(x+λd): λ L }.

6. Method of Hooke and Jeeves The method of Hooke and Jeeves performs two types of search: • Exploratory search • Pattern search

7. Method of Hooke and Jeeves Given x1, an exploratory search along the coordinate directions produces the point x2. Now a pattern search along the directionx2- x1leads to the point y. y Pattern Search x2 Exploratory search along the coordinate axes x1

8. Method of Hooke and Jeeves Another exploratory search starting from y gives the point x3. The next pattern search is conducted along the direction x3 –x2, yielding y'. The process is then repeated. y Pattern Search y' x2 x3 x1 Exploratory search along the coordinate axes

9. Summary of the Method of Hooke and Jeeves Using Line Searches As originally proposed by Hooke and Jeeves, the method does not perform any line search but rather takes discrete steps along the search directions, as we discuss later. Here we present a continuous version of the method using line searches along the coordinate directions d1 ,..., dnand the pattern direction.

10. Initialization Step • Choose a scalar ε> 0 to be used in terminating the algorithm. • Choose a starting point x1 • let y1= x1 • let k = j = 1 • Go to the Main Step.

11. Main Step • Step 1: • Let λjbe an optimal solution to the problem to: minimize f(yj +λdj) subject to: λ R yj+1= yj +λjdj • If j < n, replace j by j + 1, and repeat Step 1. Otherwise, if j = n, let xk+1=yn+1. • If ||xk+1- xk|| < ε , stop; otherwise, go to Step2.

12. Main Step • Step 2: • Let d = xk+1 – xk • Let be an optimal solution to the problem to: minimize f(xk+1+ λd) Subject to: λ R • Let y1 = xk+1+ d • Let j = 1 • replace k by k+1 • Go to Step 1.

13. Example I Consider the following problem: Minimize (x1 – 2)4+(x1 – 2x2)2 • Note that the optimal solution is (2, 1) with objective value equal to zero. At each iteration: • An exploratory search along the coordinate directions gives the points y2and y3 • Apattern search along the direction d = xk+1 – xkgives the point y1 • Except at iteration k = 1, where y1= x1 • Four iterations were required to move from the initial point to the optimal point (2, 1) whose objective value is zero. • At this point, ||x5 – x4|| = 0.045 • and the procedure is terminated.

14. Summary of Computations for the Method of Hooke and Jeeves Using Line Searches

15. Method of Hooke and Jeeves using line searches.

16. Point The Pattern Search has substantially improved the Convergence behavior by moving along a direction that is almost parallel to the valley shown by dashed lines.

17. Convergence of the Method of Hooke and Jeeves Suppose that: • f differentiable • Let the solution set Ω= { : f ( )= 0). Note:Each iteration of the method of Hooke and Jeeves consists of an application of the cyclic coordinate method, in addition to a pattern search. • Let the cyclic coordinate search be denoted by the map B and the pattern search be denoted by the map C. • If the minimum of falong any line is unique and letting α = f then α(y) < α( x ) for x Ω. • By the definition of C, α(z) ≤ α(y) for z C(y). Assuming that Λ = { x : f ( x ) < f ( x1 ) } , where x1 is the starting point, is compact, convergence of the procedure is established.

18. Method of Hooke and Jeeves with Discrete Steps The method of Hooke and Jeeves, as originally proposed, does not perform line searches but, instead, adopts a simple scheme involving functional evaluations. A summary of the method is given below:

19. Initialization Step • Let d1,...,dn be the coordinate directions. • Choose a scalar ε > 0 to be used for terminating the algorithm. • choose an initial step size, Δ ≥ ε. • Choose an acceleration factor,α>0. • Choose a starting point x1, let y1= x1, let k=j=1 • Go to the Main Step.

20. Main Step • Comparing f(yj+Δdj) and f(yj): • If f(yj+Δdj) < f(yj),the trial is termed a success; let yj+1= yj+Δdj, and go to Step2. • If f(yj+Δdj) ≥ f(yj), the trial is deemed a failure. In this case: • If f(yj -Δdj) < f(yj), let yj+1= yj-Δdj , and go to Step 2; • if f(yj-Δdj) ≥ f(yj), let yj+1= yj, and go to Step 2 .

21. Main Step • Comparing j and n • If j < n, replace jby j+1, and repeat Step 1. • Otherwise: • if f(yn+1) < f(xk) go to Step 3. • if f(yn+1) ≥f(xk) go to Step 4. • Do as follows: • Let xk+1 = yn+1 • Let y1 = xk+1+ α(xk+1 – xk) • Replace k by k+ l • Letj = 1 • Go to Step1.

22. Main Step • Compare Δ and ε • IfΔ ≤ ε, stop ; xkis the prescribedsolution. • Otherwise, replaceΔ by Δ /2. • Let y1 = xk • Let xk+1=xk • Replacek by k+ l , • Letj = 1 • RepeatStep 1.

23. Note • Steps I and II describe an exploratory search. • Step III is an acceleration step along the direction xk+1- xk. • A decision whether to accept or reject the acceleration step is not made until after an exploratory search is performed. • In Step IV, the step size Δisreduced. The procedure could easily be modified so that different step sizes are used along the different directions. This is sometimes adopted for the purpose of scaling.

24. Example II: Solve the problem using the method of Hooke and Jeeves with discrete steps Consider the following problem: Minimize (x1 – 2)4+ (x1- 2 x2 )2 In which the parameters αand Δare chosen as 1.0 and 0.2, respectively. • The algorithm start from (0.0, 3.0). • The points generated are numbered equentially. • The acceleration step that is rejected is shown by the dashed lines.

25. Method of Hooke and Jeeves using discrete steps starting from(0.0,3.0). The numbers denote the order in which points are generated.

26. The following table give a more comprehensive illustration, It summarizes the computations starting from the new initial point (2.0, 3.0). Here: • (S) denotes that the trial is a Success • (F) denotes that the trial is a Failure • At the iteration, and at subsequent iterations whenever: • If f(y3) ≥ f (xk) thenthe vector y1 is taken as xk. • Otherwise, y1= 2xk+1- xk. Note: At the end iteration k =10, the point (1.70, 0.80) is reached having an objective value 0.02. • The procedure is stopped here with the termination parameter ε= 0.1. • If a greater degree of accuracy is required, Δshould be reduced to 0.05.

27. The Summary Table of Computations for the Method of Hooke and Jeeves with Discrete Steps

29. Method of Hooke and Jeeves using line searches. The numbers denote the order in which Donets are generated.

30. Thank you all for your attention