SIGMA Workshop Part 1: Pattern-oriented Testing and Analysis

1. SIGMA WorkshopPart 1: Pattern-oriented Testing and Analysis GönençYücel, Can Sücüllü SESDYN Research Group Boğaziçi University, Istanbul

2. Similarity of dynamic behaviors • Comparing/evaluating model outcomes • Calibration • Validation • Behavior analysis • Scenario and policy analysis • Similarity in terms of what? • Pattern characteristics

3. Thinking in patterns • Problem identification/description • Oscillating inventory • Exponentially growing resource consumption • Declining sales • Model calibration and behavior validation • Sensitivity analysis • Numeric sensitivity vs. behavior sensitivity • Policy analysis

5. BATS - Can Sücüllü & Gönenç Yücel - ISDC 2014 Delft Patternwise Comparison

6. Behavior Analysis and Testing Software (BATS)

7. Overview of BATS with External Parties Import Data-series BATS ParseCommands Excel Export Data-series Receive Simulation results Controlthe Software Viewthe Results USER

8. Data Importing Methods • Excel • txt, csv, tab • Drawn by the user • Vensim results files (.vdf) • Vensim models through Model Docking Window

9. Model Docking Window of BATS • Exporting of simulation results from Vensim models Send commands; Run Sendsimulation results to Workspace in order to further analyze the model output using data analysis features Set parameter values

10. Overview of Features of BATS • Data Importing • Load From File • Model Docking Window • Draw • Data Visualization • Plot • Model Analysis • Hypothesis Tester • Behavior Space Classifier • Behavior Class Mapper * • Data Analysis • Classify • Trend • Autocorrelation • Autocorrelation Test • Spectral Density • Amplitude Estimation • Crosscorrelation • Summary Stats • Graphical Comparison * * * • Data Preparation • Split • Select • Exponential Smoothing • Moving Average • < Trend > *

11. Graphical Comparison

12. BATS Hypothesis Tester Extreme conditions are imposed parameter values Model is run and output is obtained IF hypothesized behavior <-3 FAILED IF hypothesized behavior >-3 PASSED IF multiple >-3 INCONCLUSIVE ISTS Algorithm is called

13. Behavior Space Classifier Settings

14. BATS Behavior Space Classifier Outcome Variable Parameter 2 Range 2 For each cell,a simulation run is made Parameter 1 Range 1 Model output is taken from Vensim The highest valueisentered in theClassifierGridcell ISTS Algorithm is called 25 likelihoods are obtained oscct Param1=1 Param2=9

15. Behavior Class Mapper Settings

16. BATS Behavior Class Mapper Pre-specified class is sshgr (s-shaped-growth) Foreachpoint,a simulation run is made Model output is taken from Vensim The likelihoodvalue for pre-specified class is plotted on the graph 25 likelihoods are obtained ISTS Algorithm is called Param1=8 Param2=16 -3.99

17. Identified Usage Modes of BATS • Structure-oriented behavior validity testing • Extreme condition tests • Behavior pattern validity testing • Barlas’ Multi-Step procedure and Graphical Inspection • Sensitivity analysis • Behavior pattern sensitivity with respect to parameter changes • Model calibration • Parameter calibration based on behavior patterns • Policy design • Policy parameter specification based on behavior patterns • Policy analysis • Policy structure specification based on pattern characteristics of model behaviors

18. Current version of BATS can be downloaded from the following link (Only for Windows for now. Mac version will be coming soon) • Website: http://www.ie.boun.edu.tr/labs/sesdyn/projects/bats/index.html • Download links:http://www.ie.boun.edu.tr/labs/sesdyn/projects/bats/bats_v1_11_win.ziphttp://www.ie.boun.edu.tr/labs/sesdyn/projects/bats/bats_v1_11_win.rar • Please, keep in mind that this is a beta version. • And if you give it a try, do not forget to send us some feedback that can help us to improve it.

19. Thank you… For further questions Gönenç Yücel, Assist. Prof. Boğaziçi University Industrial Engineering Dept. gonenc.yucel@boun.edu.tr