Output Analysis 2 Law and Kelton Ch.9 Steady-State Simulation

1. Output Analysis (2)(Law and Kelton Ch.9)Steady-State Simulation

2. 2 Choosing Initial Condition WHY? The measures of performance for a terminating simulation depend explicitly on the state of the system at time 0 Care must be taken in choosing appropriate initial conditions For example, to simulate a bank system between 12 noon and 1pm. (the busiest period), starting the simulation with no customer present (an empty system) will cause the results to be biased

3. 3 Choosing Initial Condition The Warm-up period To simulate a bank system, start the simulation with an empty system at 9am. and run it to 1pm. Truncate the simulation between 9am. and noon Lost 3 hours of simulated time that are not used directly in the estimate (3 hours of warm-up) If start simulating at 11am. with an empty system Only lost one hour of simulated time (1 hour of warm-up) But no guarantee the conditions in the simulation at noon will be representative of the actual conditions An alternative approach Collect data on the number of customers present in the bank at noon for several different days Start the simulation at noon using the collected data as the initial conditions for each run

4. 4 Statistical Analysis for Steady-State The Problem of the Initial Transient To estimate the steady-state mean ? = E(Y), the most serious problem is probably that E [Y(m)] ? ? for any m The technique most often suggested for dealing with this problem is called warming up the model or initial-data deletion It is to delete some number of observations form the beginning of a run and to use only the remaining observations to estimate ?

5. 5 Statistical Analysis for Steady-State Given the observations Y1, Y2, �, Ym Y(m,l) to be less biased than Y(m), since the observations near the �beginning� of simulation may not very representative of steady-state behavior due to the choice of initial conditions The question is: How to choose the warm-up period (or deletion amount) l?

6. 6 Statistical Analysis for Steady-State Warm-Up Period To pick l and m such that E [Y(m, l)] ? ? If l and m are chosen too small, then E [Y(m, l)] may be significantly different from ? If l is chosen larger than necessary, then Y(m, l) will probably have an unnecessarily large variance In general, it is very difficult to determine l from a single replication due to the inherent variability of the process Y1, Y2, � (L&K Fig. 9.8)

7. 7 Statistical Analysis for Steady-State The simplest and most general technique to determine l is a graphical procedure due to Welch (1981, 1983) 1. Make n replications (n?5), each of a large length m. Let Yji be the ith observation from the jth replication (L&K Fig. 9.5) 2. Let 3. Define the moving average to smooth out the high-frequency oscillations: 4. Plot Yi(w) for i = 1, 2, �, m - w and choose l to be that value of i beyond which Y1(w), Y2(w), � appears to have converged

8. 8 Statistical Analysis for Steady-State Recommendations on choosing n, m, and w Initially, make n = 5 or 10 replications with m as large as practical m should be much larger than l and large enough to allow infrequent events (e.g., m/c breakdowns) to occur a reasonable number of times Plot Yi(w) for several values of the window w and choose the smallest value of w for which the corresponding plot is �reasonably smooth� Use the plot to determine l If no value of w in step 3 of Welch�s procedure is satisfactory, make 5 or 10 additional runs of length m. Repeat step 2 using all available replications

9. 9

10. 10 Inherent variabilities Example: A small factory consisting of a m/c center and an inspection station produces Ni parts in the ith hour To determine l so that we can estimate the steady-state mean hourly throughput ? = E(N) Make n = 10 replications with each m = 160 hours Plot the averaged process Ni for i = 1, �, 160 (Fig. 9.10) Plot the moving average Ni(w) for w = 20 and w = 30 (Fig. 9.8 (a) & (b)) where l = 24 hours for w = 30 Note that it is probably better to choose l too large rather than too small, since we want to have E(Yi) close to ? for i > l

11. Inherent variabilities make it difficult to assess initial warm-up and entry to steady-state 11

12. Applying moving average, w=20 12

13. Applying moving average, w=30 13

14. 14 Other Approaches for Means Two general strategies for means Fixed-Sample-Size Procedures A single simulation run of an arbitrary fixed length is made, and then one of a number of available procedures is used to construct a confidence interval from the available data Sequential Procedures The length of a single simulation run is sequentially increased until an �acceptable� confidence interval can be constructed There are several techniques for deciding when to stop the simulation run

15. 15 Fixed-Sample-Size Procedures Six fixed-sample-size procedures (Table 9.6) Replication/deletion Batch means Autoregressive Spectral Regenerative Standardized time series

16. 16

17. 17 Fixed-Sample-Size Procedures Replication/deletion As discussed previously, it is based on n independent �short� replications of length m observations Tends to suffer from bias in the point estimator The five other approaches Based on one �long� replications Tend to have a problem with bias in the estimator of the variance of the point estimator

18. 18 Replication/Deletion Approach for Means Reasons If properly applied, this approach gives reasonably good statistical performance The easiest approach to understand and implement Apply to all types of output parameters Easy to estimate different parameters for the same simulation model Can be used to compare different system configurations

19. 19 Replication/Deletion Approach for Means Similar to that for terminating simulation except that only those observations beyond the warm-up period l in each replication are used to form the estimates Let Xj be given by where n� = number of replications m� = length of observations (much larger than l) An approximately unbiased point estimator is given by

20. 20 Fixed-Sample-Size Procedures Batch Means Based on a single long run, so it has to go through the �transient period� only once Make a simulation run of length m and then divide the resulting observations into n batches of length k (assume that m = nk) Let Yj(k) be the sample (or batch) mean of the k observations in the jth batch If the batch size k is large enough, it can be shown that the Yj(k)�s will be approximately uncorrelated If the batch size k is too small, the Yj(k)�s will possibly be highly correlated and S2(n)/n be a severely biased estimator of variance

21. 21 Fixed-Sample-Size Procedures Autoregressive Assume that the process is covariance stationary with mean and can be represented by the pth-order autoregressive model

22. 22 Fixed-Sample-Size Procedures Autoregressive For large m, an estimate of variance and a confidence interval for mean are given by

23. 23 Fixed-Sample-Size Procedures Spectrum Analysis Also assume that the process is covariance stationary with mean, then it is possible that

24. 24 Fixed-Sample-Size Procedures Regenerative To identify random times at which the process probabilistically �starts over�, i.e., regenerates, and to use these regeneration points to obtain independent random variables to which classical statistical analysis can be applied to form point and interval estimates for mean The difficulty with using this method in practice is that real-world simulations may not have regeneration points, or the expected cycle length may be so large that only a very few cycles can be simulated

25. 25 Fixed-Sample-Size Procedures Regenerative Suppose that we simulate the process for n� regeneration cycles, resulting in the following data: Z1, Z2, �, Zn� and N1, N2, �, Nn� A point estimator for mean confidence interval is given by

26. 26 Fixed-Sample-Size Procedures Standardized Time Series Suppose that we make one simulation run of length m and divide into n batches of size k The major source of error for this method is choosing the batch size k too small If m is large, then the sample mean of the k observations will be approximately normally distributed with mean ? and variance ?2/m, where

27. 27 Fixed-Sample-Size Procedures Standardized Time Series Let For large k, we can treat A confidence interval for mean is given by

28. 28 Fixed-Sample-Size Procedures Conclusions If the total sample size m (or n�) is too small, the actual coverage of all these procedures may be considerably lower than desired since a steady-state parameter is defined as a limit as the length of the simulation goes to infinity The �appropriate� choice of m (or n�) would appear to be extremely model-dependent and thus impossible to choose arbitrarily For m fixed, the methods f batch means, standardized time series, and spectrum analysis will achieve the best coverage for n and f small

29. 29 Sequential Procedures Sequentially determine the length of a single simulation run needed to construct an acceptable confidence interval for the steady-state mean We might want to determine a run length large enough to obtain an estimate of mean with a specified absolute error ? or relative error ? See L&K for detail discussions about some recommended procedures