Chapter 6

1. Chapter 6 Statistical Analysis of Output from Terminating Simulations

2. Statistical Analysis of Output from Terminating Simulations • Random input leads to random output (RIRO) • Run a simulation (once) — what does it mean? • Was this run “typical” or not? • Variability from run to run (of the same model)? • Need statistical analysis of output data • From a single model configuration • Compare two or more different configurations • Search for an optimal configuration • Statistical analysis of output is often ignored • This is a big mistake – no idea of precision of results • Not hard or time-consuming to do this – it just takes a little planning and thought, then some (cheap) computer time Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

3. Output Analysis Output analysis is concerned with • Designing replications Obtain most reliable info with minimum number of replications and minimum run length. • Computing statistics Point and confidence interval estimation Size and independency issues • Presenting them in a textual and graphical format. Aim is to understand the system behavior and generate predictions for it! Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

4. Time Frame of Simulations • Terminating: Specific starting, stopping conditions • Run length will be well-defined (and finite) • Steady-state: Long-run (technically forever) • Theoretically, initial conditions don’t matter (but practically they usually do) • Not clear how to terminate a simulation run • This is really a question of intent of the study • Has major impact on how output analysis is done • Sometimes it’s not clear which is appropriate Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

5. Model 6.1 • Same as Model 5.3 Number of trunk lines=26 No additional staff during 5-8 hrs. • 10 runs are made For terminating case, make IID replicationsRun>Setup>Replication Parameters: Number of Replications=10 Check both boxes for Initialize Between Replications • Outputs are saved to .dat files Statistics Module, Type=output, Data file name= Filename.dat Asli Sencer

6. Outputs of Model 6.1 • Category Overview report will have some statistical-analysis results of the output across the replications Asli Sencer

7. Output Precisionin Model 6.1 • This information (except standard deviation) is in Category Overview report • If > 1 replication specified, Arena uses cross-replication data as above • For otherconfidence levels or graphics – Output Analyzer Asli Sencer

8. Confidence Interval Estimation Notation: number of replications sample mean sample standard deviation critical value for significance level (1-α)x100% Confidence Interval (CI): for for ,

9. Interpretation of a Confidence Interval • A CI is an intervalwith random (data-dependent) endpoints that’s supposed to have stated probability (1-α) of containing, or covering, the expected values • “Target” expected value is a fixed, but unknown, number • Expected value = average of infinite number of replications • A CI is notan interval that contains, say, 95% of the data generated by simulation outputs. • That’s a prediction interval … useful too, but different • 95% CI means: If we make infinitely many batch runs of size and construct C.I. for α=5%, 95% of the time this interval will contain the true (but unknown) population parameter. Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

10. Required Number of Replications to Achieve a Certain Precision • Half-width = • We prefer confidence intervals with smaller half width— for higher precision • Question: How many replications are needed to get the required (better) precision? We can’tcontrol t or s. We may justincrease n — how much? • Trial and error (now) • Approximate number for acceptable precision (below) • Sequential sampling (Chapter 11) Want this to be “small,” say < h where h is prespecified Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

11. Half Width and Number of Replications • Let required half-width, solve for • Not really solved for n (t, s depend on n) • Approximation: • Replace t by z, corresponding normal critical value • Pretend that current s will hold for larger samples • Get • Easier but different approximation: s = sample standard deviation from “initial” number n0 of replications n grows quadratically as h decreases. h0 = half width from “initial” number n0 of replications Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

12. Number of Replications Needed • If we require h=$250 rather than $812 for total cost, replications are needed. Asli Sencer

13. Model 6.2 • 110 Runs • 95% CI for the total cost= 22,175.19 +- 369.54, half width=369.54>250! Required accuracy is still not met, why? As more replications are made, in formula, does not change, increases; but , the sample standard deviation might increase. So might decrease. • 95% CI for the %rejected calls= 11.73.96 +- 0.51, accuracy is quite satisfactory! Asli Sencer

14. Model 6.3 • 1000 Runs-as a trial • Save the output to Total cost.dat • Open Output analyzer as a separate application File>Data File>Export Export binary data in .dat file to a plain ASCII text file and save. • Open Arena Input Analyzer Plot the histogram of the Total Costs Asli Sencer

15. Histogram of 1000 Total Cost Values • Since Total Cost values is a sum, law of large numbers apply. We see that the distribution approaches normal as the number of replications increase! • Same is true for average statistics due to central limit theorem. • It is not true for extreme value statistics like maximum or minimum. Asli Sencer

16. ConfidenceIntervals(cont’d) • Usual formulas assume normally-distributed data • Never true in simulation • Might be approximately true if output is an average, rather than an extreme • Central limit theorem • Issues of robustness, coverage, precision – details in book Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

17. Comparison of AlternativesStatistical Hypothesis Test is the mean performance of system i Reject Ho if is significantly large or small, i.e., performance of system 1 is significantly different than system 2! Here: : Total Cost of Base Model (110 observations) : Total Cost of Alternative Model (110 observations) Chapter 5 – Detailed Modeling and Terminating Statistical Analysis

18. Comparing Two Scenarios • Base Scenario: Model 6.4 (Same as in Model 5.3) -110 runs -26 Trunk Lines, No New Staff between 12:00-16:00 • Alternative scenario: Model 6.4 (More-resources scenario) -110 runs -29 Trunk Lines, (Change the capacity from 26 to 29) -Hire three for each of Larry, Moe, Curly, Hermann and Sales Resources. (Change these variables from 0 to 3) • Tradeoff is between increased salary cost but decreased excess waiting costs. Will the total costs decrease? • Percent Rejected calls will decrease, but how much? Asli Sencer

19. Comparison of Scenarios • Runs both models for 110 times. • Statistics Data Module Save output files –BaseCase.dat or -MoreResources.dat. • 95% CI for total costs are Base model: 22,175.19 +- 369.54=[21,805, 22,544] Increased resources: 24,542.82 +- 329.11=[24,213, 24,871] Intervals do NOT overlap, hence Total Costs are significantly different at 5% significance level. • 95% CI for percent rejected are Base model: 11.74 +- 0.51=[11.23, 12.25] Increased resources: 1.73 +- 0.31=[1.42, 2.04] Intervals do NOT overlap, hence Percent Rejected are significantly different at 5% significance level. Asli Sencer

20. Arena Output Analyzer • Separate application in Arena • Operates in output files (.dat) generated by Arena through the Statistics data module • Data in .dat file is in binary format to be opened by Arena Output Analyzer only! • Provides confidence intervals on expected output statistics as also appear in Arena output reports. • Provides statistical comparison of two scenarios, and others. Asli Sencer

21. Comparison of Scenarios with Arena Output Analyzer • Open Output Analyzer • Select File>New to open a data group, i.e., list of .dat files • Add TotalCost-BaseCase.datTotalCost-MoreResources.datPercentRejected-BaseCase.datPercentRejected-MoreResources.dat • Can save this data group as .dgr file to refer easily afterwards • Analyze>Compare Means Add each pair of comparisons by choosing ‘lumped’ so that all 110 values are considered in the analysis Asli Sencer

22. Hypothesis Tests • Ho: Mean TC of base case = Mean TC of more resources Ha: Mean TC of base case ≠ Mean TC of more resources • Ho: Mean % rejected of base case = Mean % rejected of more resources case Ha: Mean % rejected of base case ≠ Mean % rejected of more resourcescase Asli Sencer

23. Output Report-Compare Means Confidence interval on difference misses 0, so conclude that there is a (statistically) significant difference between the base model and the alternative at α=5% Asli Sencer

24. Evaluating Many Scenarios with Process Analyzer • Separate application in Arena • Allows making multiple pairwise scenario comparisons at a time. • PAN operates on Arena program files with .p extension, generated when .doe model is run. • A PAN scenario includes a program file, a set of values for the input controls (decision variables in the form of variables and resources), a set of output responses. • A PAN project is a collection of such scenarios that can be saved by .pan extension for future reference. Asli Sencer

25. Development of a PAN Project • Use Model 6.5 110 runs Output data files are deleted since they will be useless in PAN. • Open a PAN project File > New, File > Open • Add a new scenario. Double click on the raw Name=Base Case, Program File=Model 6.5.p Replications=110 • Add contols Right click in this line OR Insert > Control Under Resources: The capacity of trunk line Under User Specified: New Tech 1, New Tech 2, New Tech 3, New Tech All, New Sales • Add responses Right click on this line OR Insert > Responses Under user specified: Total Cost, Percent Rejected Asli Sencer

26. New Scenarios • Suppose you have $1360/week to spend on all additional resources. To which of the six expandable resources should you allocate the new money? • Then following 6 alternative scenarios apply in addition to Base Case. 13 more trunk lines ($98 each) 4 more tech 1,2,3 people ($320 each) 3 more tech all people ($360 each) 4 more sales people ( 340 each) • Run the scenarios Check the scenarios to run Run > Go OR play button OR F5 function key AsliSencer

27. PAN screen Asli Sencer

28. Generating Reports for Multiple Comparison in PAN • Insert > Chart OR right click on a response column. Chart type=Box whiskers Check Identify Best Scenarios box Select ‘smaller is the better’ • Red boxes are significantly better than blue ones at 5% significance level. • To decrease the half width of a scenario, increase the number of replications of that specific one. • Error tolerance is a positive value that represents an amount small enough that you don’t care if the selected scenarios are actually inferior to the true best one by at most this amount. A positive error reduce the number of selected scenarios at the risk of being off by a little bit. Asli Sencer

29. A PAN Report Asli Sencer