AutoSimOA : A Framework for Automated Analysis of Simulation Output

1. AutoSimOA:A Framework for Automated Analysis of Simulation Output Stewart Robinson (stewart.robinson@warwick.ac.uk), Katy Hoad, Ruth Davies Funded by EPSRC and SIMUL8 Corporation

2. The Warwick Simulation Research Group DES 7 members of staff 2 research fellows 4 PhD students Focus on the practice and application of simulation methods SD ABS

3. The Warwick Simulation Research Group Recent/current projects: • Comparison of DES and SD model building • Agent based modelling of social networks • Effect of model reuse on learning • Conceptual modelling for DES • Agent based modelling for service systems • Human interactions in supply chains • Simulation and lean in the health service • …

4. The Problem • Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts. • Simulation software generally have very limited facilities for directing/advising user how to run the model to get accurate estimates of performance. • With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

5. Aim • To develop an automated output analysis system that can be implemented in commercial simulation software with a view to improving the use of simulation, particularly by non-expert simulation users.

6. More formally… • To develop an automated procedure that obtains unbiased estimators (of specified precision) for the population mean and variance (μ and σ2 respectively) for one or more simulation output statistics.

7. Transient Simulation Output

8. Steady-State Simulation Output

9. 3 Main Decisions • How long a warm-up is needed? • How many replications should be run? • How long a run length is needed?

10. Work Carried Out for AutoSimOA Project • Classification of different model types and output data properties. • Extensive testing of replications algorithm. • Literature review of (44) warm-up methods. • Tested MSER-5 to destruction using over 3000 data sets. • Literature review of batch means methods. • Development of AutoSimOA.

11. Enter Analyser AutoSimOA Replications or a single run? Replications Single run Warm-up? Warm-up? No Yes Yes No Replications Calculator Warm-up Analyser Single Run Analyser EXIT Analyser

12. Replications Calculator

13. Confidence Interval Method with ‘Look-ahead’ Precision ≤ 5% Precision > 5% Precision ≤ 5% 95% confidence limits Cumulative mean, f(kLimit) Nsol2 + f(kLimit) Nsol1 Nsol2

14. Warm-up Analyser • MSER-5 most promising method for automation • Performs robustly and effectively for the majority of data sets tested. • Not model or data type specific. • No estimation of parameters needed. • Can function without user intervention. • Quick to run. • Fairly simple to understand.

15. Dealing with Initialisation Bias Warm-up Period: MSER-5 Heuristic Minimises mean squared error of output data. Performs analysis on batch mean data – batch size of 5. MSER-5 value calculated as follows:

16. Dealing with Initialisation Bias Warm-up Period: MSER-5 Heuristic

17. Heuristic framework around MSER-5 Adaptation in to a sequential procedure: • Iterative procedure for procuring more data when required. • ‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated. • Graphical feedback to user.

18. Single Run Analyser There are 3 possibilites: • User wants a mean estimate with a CI of a specific precision. • User has a specific run-length in mind & wants a mean estimate with a valid CI at end of run (i.e. no precision requirement). • User neither requires a specific precision nor has a set run length in mind.

19. SINGLE RUN ANALYSER Use set run-length? NO YES Batch Means Calculator Run-length Calculator Abort ASAP3 LABATCH2 (Steiger et al, 2005) (Fishman, 1998)

20. Example Implementation of AutoSimOA Data: • ‘user support model’ - simulates calls received, processed and actioned at an IT support help desk (Robinson, 2001). • Output of interest = average time calls spend in the system. • Steady-state output with a substantial initial bias. • True steady-state mean estimated as 2,269 mins (using a long run with 54,000 data points).

21. Implementation Issues • Output data type – What should and should not be analysed? • Cumulative values, extreme values • Time or entity data • Multiple outputs • Analyse all outputs of interest to user. • Multiple scenarios • Run for all scenarios? Run for just the base case? • Issues regarding run length with ASAP3.

22. Automation Issues • Generation of more data when required. • Run simulation from present termination point. • Single run vs replications. User involvement: • User decision of ‘what to do’- based on knowledge of nature of model & output. • Warm-up needed? Multiple replications? • One run? Length of run for replications? • Determining if recommendations are reasonable • Graphical aids.

23. Limitations of AutoSimOA • Not directly able to handle cyclic data. • Unable to analysewarm-up for transient output data subject to initialisation bias. • Only performs an analysis on the mean and variance of the output statistics of interest. • Median, mode, quantiles,… • Provides no facilities for scenario analysis. • Ranking and selection, optimisation,…

24. ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project. Stewart Robinson Warwick Business School Brunel DISC Seminar December 2009