Experiment Design

1. Experiment Design • Suppose you want to do some experiments to improve a manufacturing process you think may be influenced by a dozen different factors • What do you really want from the experiment – what does “optimal” mean? • How can you most economically get the information you want? David Drain

2. An Experiment • Three “factors” (potential influences) • Seven “runs” – the process is run at this combination of settings and data is collected David Drain

3. Noise Variables and Process Targets • A robust process target gives good results regardless of noise variables • Control-noise interaction terms measure robustness David Drain

4. Constraints in Industrial Design Problems • Number of runs • Mandatory center points • Blocks • Constraints on control variables David Drain

5. Contour Plot of PEV • Plot of prediction error variance for modified CCD with 2 control and 2 noise variables: David Drain

6. Sensitivity of a Classic Design FCC with 23 runs No noise Noise parameters of magnitude 0.18 David Drain

7. Design Discovery Methods • Try the classic designs (CCD,…) • Exchange methods • Choose candidates, then an algorithm (DETMAX, for example) selects a “best” subset • Candidate set must include optimal runs to get optimal design • Selecting too many candidates makes the algorithm computationally intractable • Heuristic methods David Drain

8. Hybrid Heuristics • A hybrid heuristic is some combination of basic heuristics • Starting a GA with a random population is a simple hybrid: scattersearch followed by GA • Talbi (2002) gives a complete taxonomy of hybrid heuristics David Drain

9. The No Free Lunch Theorem • Scattersearch is just as good as an efficient hillclimbing algorithm, if their performance is averaged over all possible problems. • No heuristic can be applied successfully to all problems: the heuristic must be chosen to match the problem being solved. “Roughly speaking, the average performance of any pair of algorithms across all possible problems is identical.” Wolpert and Macready (1997) David Drain

10. Genetic Algorithm • Based on evolution: genetic material from two parents is combined to produce offspring, and some mutation is allowed Parent A Parent B Child Crossover David Drain

11. Simulated Annealing • Based on physical annealing processes • Early in the process, while at a high “temperature”, relatively large changes from local optimum are allowed. • Disadvantageous changes are allowed with a probability inversely related to temperature • Requires multiple starting points because of tendency to become trapped at local optima David Drain

12. GASA: Genetic Algorithm – Simulated Annealing Hybrid • Integrated hybrid of GA and SA • Exploration with temperature-controlled GA • Exploitationwith SA within each GA step • Discrete changes with GA (entire runs) • Continuous changes with SA • Temperature parameters tunable for computational efficiency and effectiveness David Drain

13. GASA Algorithm David Drain

14. GASA Initialization • Initialize population from candidate list • Read test vectors, or generate random test vectors • Define objective function • Desirability functions are supported • Read starting parameters • Define rules for parameter modification by iteration David Drain

15. GASA Parameters • Number of iterations • Population size • Number of elites • Cross-breeding requirements Iteration Dependent Fixed • Roulette rate • Mutation rate • Number of test vectors used for objective function evaluation • Distance and probability rules for SA David Drain

16. Using GASA Effectively • Parameter choices have a significant effect on algorithm effectiveness and computational efficiency • Some parameter settings can be determined through the use of designed experiments (these must estimate interactions) • Ideal parameter values seem to depend on the stage of optimization: • Early in the process, high mutation rates make sense • Late in the process, more intense SA may be needed • Dynamic parameter adjustment is a necessity David Drain

17. Using GASA Effectively • Use the best starting points available • Ensure sufficient genetic diversity • Unintentional cloning occurs otherwise • Large populations and forced crossbreeding enhance genetic diversity • Choose discriminating objective functions • No population member should achieve a perfect score, and most members should not receive a failing score • Objective function parameters might need to be refined as optimum is approached David Drain

18. GASA Results • GASA was validated on a problem recently solved by Borkowski (2003) • D-optimal design for a quadratic model in a square region using six runs • GASA can improve the best known designs • Greatest benefits are in slope estimate variance reduction and rotatability, rather than prediction error variance reduction • Smaller designs exhibit the most improvement David Drain

19. Rotatability Improvement David Drain

20. Future Research • Parameter setting optimization • Dynamic and adaptive parameter determination • “Neighborhood” definition (for SA) based on design properties rather than Euclidean distance • Find alternatives to simulated annealing for exploitation mechanism • Simple hill-climbing did not work well • Gravitating to areas of high variance did not work well David Drain

21. References • Borkowski, J.J. (2003). "Using a Genetic Algorithm to Generate Small Exact Response Surface Designs". Journal of Probability and Statistical Science 1, pp. 65-88. • Borror, C. M.; Montgomery, D. C. and Myers, R. H. (2002). “Evaluation of Statistical Designs for Experiments Involving Noise Variables”. Journal of Quality Technology, 34, pp. 54-70. • Box, G. E. P. and Wilson, L. B. (1951). “On the Experimental Attainment of Optimum Conditions”. Journal of the Royal Statistical Society B, 13, pp. 1-45. • Forouraghi, B. (2000). "A Genetic Algorithm for Multiobjective Robust Design". Applied Intelligence 12, pp. 151-161. • Haines, L. M. (1987). "The Application of the Annealing Algorithm to the Construction of Exact Optimal Designs for Linear-Regression Models". Technometrics 29, pp. 439-447. • Hamada, M.; Martz, H.F.; Reese,C.S. and Wilson, A.G. (2001). "Finding Near-Optimal Bayesian Experimental Designs via Genetic Algorithms". The American Statistician 55, pp.175-181. • Heredia-Langner, A.; Carlyle, W.M.; Montgomery, D.C.; Borror, C.M. and Runger, G.C. (2003). “Genetic Algorithms for the Construction of D-Optimal Designs”. Journal of Quality Technology, 35, pp. 28-46. • Kragelund, L. V. (1997). "Solving a Timetabling Problem using Hybrid Genetic Algorithms". Software- Practice and Experience, 27, pp. 1121-1134. David Drain

22. References (cont.) • Li, W. D.; Ong, S. K. and Nee, Y. C. (2002). "Hybrid genetic algorithm and simulated annealing approach for the optimization of process plans for prismatic parts". International Journal of Production Research, 40, pp. 1899-1922. • Myers, R. H.; Khuri, A. I. and Vining, G. (1992). "Response Surface Alternatives to the Taguchi Robust Parameter Design Approach". American Statistician, 46, pp. 131-139. • Parkinson, D.B. (2000). "Robust Design Employing a Genetic Algorithm". Quality and Reliability Engineering International, 16, pp. 201-208. • Taguchi, G. (1986). Introduction to Quality Engineering: Designing Quality into Products and Processes. Kraus International Publications, White Plains, NY. • Talbi, E.-G. (2002). "A Taxonomy of Hybrid Metaheuristics". Journal of Heuristics 8, pp. 541-564 • Welch, W.J. (1982). "Branch-and-Bound Search for Experimental Designs Based on D Optimality and Other Criteria". Technometrics 24, pp. 41-48. • Wolpert, D.H. and Macready, W.G. (1997). "No Free Lunch Theorems for Optimization". IEEE Transactions on Evolutionary Computation, 1, pp. 67-82. David Drain