Methods For Selecting The Best System Goldsman, Nelson, & Schmeiser

1. Methods For Selecting The Best SystemGoldsman, Nelson, & Schmeiser 937814 林蒼威 2005/4/14

2. References • Horrace W. C. & Keane T. P., 2004, “Ranking and selection of motor carrier safety performance by commodity”, Accident Analysis and Prevention 36, pp. 953-960 • Alrefaei M. H. & Alawneh A. J., 2004, “Selecting the best stochastic system for large scale problems in DEDS”, Mathematics and Computers in Simulation 64, pp. 237-245 • Yang W. & Nelson B. L., 1991, “Using common random numbers and control variates in multiple-comparison procedures”, Operation Research 39 (4), pp. 583-591

3. Agenda • Abstract • Airline-Reservation System • Interactive Analysis • Ranking and Selection • Multiple Comparisons • Q&A

4. Abstract • Three method • Interactive Analysis • Ranking and Selection • Multiple Comparisons • Aspects • Assumption、implementation、advantage and disadvantage • Example • Airline reservation system

5. Motivation • There are three systems, the average TTF are so which one is the best system ?

6. Strategy(1) • There are many statistic strategies using in selecting best system. • When n is fixed, the statistic strategies is to determine the value of d, then using α,βvalue to evaluate the performance of selecting best system. • If n can be determined, many methods may be applied to select best system, such as multiple comparison, ranking and selection, interactive analysis, etc.

7. Strategy(2) Interactive Analysis Ranking & Selection Multiple Comparison …

8. 1. Airline-Reservation System • Consider k=4 different systems • Objective is maximize E[TTF] • From experience, E[TTF] roughly 100,000 minutes for all four systems • Indifference zone is 3000 minutes

9. 2. Interactive Analysis • An estimation approach It considers 4 point estimators for E[TTF]’s and estimates their standard errors • The goal is a vague, but well-founded, sense of confidence in the selected system • IA here suppresses the explicit confidence-interval logic

10. 2.1 The Method • m=number of micro-replication b=number of macro-replication; n=bm • The point estimator for μi，i=1,2,3,4 (the E[TTF] of system i)： • With associated sample variance of the macro-replication estimators：

11. 2.2 The Assumptions • People choose αarbitrarily and small b &α can lead to large t-values • Choose 10<b<30 is often wise (Schmeiser) • If the value of b is reasonably large, the effects of sequential sampling are negligible

12. 2.3 The Example • Initial run is designed just to gain a sense of the magnitude of the required production experiment in terms of time per replication and number of replications • System： standard error (se)=21000 (b=m=5) indifference value=3000 So the se will need to drop to at most 1500 This means that the “worst-case” is 5000 replications

13. 3. Ranking & Selection • To select the best system from a set of competing systems • The probability of a correct selection will be at least some user-specified value (The normal means procedure of Rinott)

14. 3.1 The Method(1/3) • Ordering： The two best： • If is very small, less than δ>0 ->It wouldn’t matter which one we chose as best (δ=3000) • We take P*=0.9 in our example ->P (correct selection) ≥ P*=0.9

15. 3.1 The Method(2/3) • The first-stage sample means and sample variances • The sample variances are used to determine the number of macro-replications which must be taken in the second stage

16. 3.1 The Method(3/3) • So additional observations must be taken • Finally we get grand means and select the system with largest

17. 3.2 The Assumptions • The macro-replication estimators, from the ith system are assumed to be i.i.d. with expectation • If the number of micro-replications m is large enough ->CLT yields approximate normality for the macro-replication estimators

18. 3.3 The Example • Sequence of experiments： 1. Debugging Experiment 2. Pilot Experiment 3. Production run

19. 3.3.1 Pilot Experiment • Intend to be used as the first stage of sampling • b0=20

20. 3.3.2 Final Result • For the case k=4, P*=0.9 -> h=2.720 (from Wilcox) • For system 2, we needed to take 465 additional macro-replications in stage two • We are at least 90% sure that we have made the correct selection ( >δ=3000)

21. 4. Multiple comparisons • There is no distinction between micro and macro-replication Lower endpoint is 0, then system i is the best system Upper endpoint is 0, then system i is bad system

22. 4.1 Assumption • Data from one-way analysis of variance • Use different random number seed to generate sample, so the relationships between samples are independent • Sample from normal and common variance

23. 4.2 Pilot experiment (Batch mean) • Because n replications need to be large that samples may be normally distributed, so use batch mean method to generate sample. • Using batch mean will tend to form the normal distribution, but the drawback is the lose of degree of freedom

24. 4.3 Pilot experiment • n=200 ,b=40 and m=5 • Since all system contained 0 no system could be declared to be the best. • S=92449, d=2.078,n=8100 for final result. Let half interval of CI≤ δ

25. 4.4 Final result • Using Bartlett’s test for equality variance, the result is different, so set b1=b2=b3=100, and b4=150 let variance are same. System 1 is the best.

26. Method Comparison(1/2)

27. Method Comparison(2/2)

28. Q & A Thanks for your listening!!!