Automating the Analysis of Simulation Output Data

1. Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006

2. Outline The problem A prototype automated output Analyser Findings from prototype Analyser The AutoSimOA Project Current work - • Collecting and characterising real and artificial models

3. The Problem Prevalence of simulation software: ‘easy-to-develop’ models and use by non-experts. Simulation software generally have very limited facilities for directing/advising on simulation experiments. Main exception is directing scenario selection through ‘optimisers’. With a lack of the necessary skills and support, it is highly likely that simulation users are using their models poorly.

4. The Problem Despite continued theoretical developments in simulation output analysis, little is being put into practical use. • There are 3 factors that seem to inhibit the adoption of output analysis methods: • Limited testing of methods • Requirement for detailed statistical knowledge • Methods generally not implemented in simulation software (AutoMod/AutoStat is an exception) A solution would be to provide an automated output ‘Analyser’.

5. Simulation model Output data Analyser Warm-up analysis Obtain more output data Use replications or long-run? Replications analysis Run-length analysis Recommendation possible? Recommend- ation A Prototype Analyser • Masters Project (3 students). • The Analyser looked at: • Warm-up • Run-length • Number of replications • Scenario analysis could be added.

6. A Prototype Analyser A prototype Analyser has been developed in Microsoft Excel. At present it links to the SIMUL8 software, but it could be used with any software that can be controlled from Excel VBA.

7. Illustration: Warm-up Load Analyser into Excel. Enter name of SIMUL8 model. Specify initial number of replications and run-length to use.

8. Illustration: Warm-up

9. Illustration: Replications

10. Findings from Prototype Analyser It is possible to link an Automated Analyser in Excel to a simulation software tool. This was just a proof of concept. • Key issues to address: • More thorough testing of output analysis methods for their accuracy and their generality. • Adaptation of methods to sequential procedures and to minimise the need for user intervention.

11. The AutoSimOA Project A 3 year, EPSRC funded project (GR EP/D033640/1) in collaboration with SIMUL8 Corporation. • Objectives • To determine the most appropriate methods for automating simulation output analysis • To determine the effectiveness of the analysis methods • To revise the methods where necessary in order to improve their effectiveness and capacity for automation • To propose a procedure for automated output analysis of warm-up, replications and run-length • Only looking at analysis of a single scenario

12. The AutoSimOA Project • CURRENT WORK: • Literature review of warm-up, replications and run-length methods. • Development of artificial data sets (Auto-Regressive; Moving average; M/M/n/p Queues…) • Collection of ‘real’ simulation models.

13. Use models / data sets: • Provide a representative and sufficient set of models / data output for use in discrete event simulation research. • Use models / data sets to test the chosen simulation output analysis methods in the AutoSimOA Project..

14. Auto Correlation Spread round mean In/out of control Terminating Group B Non-terminating Cycling/Seasonality Normality Transient Steady state Trend

15. Output data characteristics • Model characteristics • Deterministic or random • Significant pre-determined model changes (by time) • Dynamic internal changes i.e. ‘feed-back’ • Empty-to-empty pattern • Initial transient (warm-up) • Out of control trend ρ≥1 • Cycle • Auto-correlation • Statistical distribution

16. ARTIFICIAL MODELS • Create simple simulation models where theoretical value of some attribute is known. E.g. M/M/1: mean waiting time. • Create simple simulation models where value of some attribute is estimated but model characteristics can be controlled. E.g. Single item inventory management system: Number-in-stock. • Create data sets from known equations, which closely resembles real model output, with known value for some specific attribute. E.g. AR(1) with Normal errors Create different output types Transient Steady state cycle Trend + Initial transient (warm-up) Steady state

17. Example artificial models: 1.E4 ~ Erlang(4) / M / 1 Queue mean 1.8 Traffic Intensity = 0.8

18. Initial Bias Functions: Exponential Mean shift 2 Under Damped oscillations Example artificial models: 2.Auto-Regressive (2) series

20. e.g. Call centre: percentage of calls answered within 30 secs e.g. Production Line Manufacturing Plant: through-put / hour REAL MODELS Models created in “real circumstances” that cover each general type of model and output encountered in real life modeling. e.g. Fast Food Store: average queuing time e.g. Swimming Pool complex: average number in system Steady State Transient Steady State Cycle With or without warm-up Trend

21. Example ‘ real ’ models: 1.Argos – Number of customers in queue to pay Stochastic model with changing arrival rates. Empty to empty; transient; autocorrelated; non-normal output.

22. Example ‘ real ’ models: 2. Leggings Manufacturing Plant – Through-put / hour Stochastic model. Steady state with warm-up; not autocorrelated; normal output.

23. Example ‘ real ’ models: 3. Sanitory Towel Packing Plant – Through-put / hour Stochastic model with changing productivity in work stations. Steady state daily cycle. Series of means of each cycle: autocorrelated; non-normal output.

24. The AutoSimOA Project Create a standard set of model outputs for testing output analysis methods. • We will use this representative and sufficient set of models/output when: • determining the most appropriate methods for automating simulation output analysis • determining the effectiveness of the analysis methods • revising the methods where necessary in order to improve their effectiveness and capacity for automation • In order to propose a procedure for automated output analysis of warm-up, replications and run-length.

25. Thank you for listening. Automating the Analysis of Simulation Output Data Stewart Robinson, Katy Hoad, Ruth Davies OR48, September 2006