Analyzing Agent-Based Models: Practical Strategies for Understanding and Testing ABMs

1. MIS 643 Agent-Based Modeling and Simulation (ABMS) Bertan Badur badur@boun.edu.tr Department of Management Information Systems Boğaziçi University

2. Model Analysis • Chapter 21-23, of Agent-Based and Individual-Based Modeling: A Practical Introduction,by S. F. Railsback and V. Grimm

3. Outline • Chapter 21: Introduction to Part IV • Chapter 22: Analyzing and Understanding ABMs • Chapter 23: Sensitivity, Uncertainty and Robustness Analysis • Chapter 24:

4. Chapter 21: Introduction to Part IV • 21.1 Objectives of Part IV • 21.2 Overview of Part IV

5. 21.1 Objectives of Part IV • Testing – checking whether a model or submodel is correctly implemented and does what it is supposed to do • Analysing a model: trying to understand what a model does • Understanding not automatic • transfer understanding from model to real system • from begining of modeling cycle • submodels or simple models • POM for sturucture, theory, calibration • Full models • “freeze” design at some point • understand how it works and behave

6. not too soon • once the model • key processes • represent real system reasonably • version number 1.0 • after that two or three versions is likely • Programming and testing easy • What is science? • relation between model and real system – POM Part III • analyse throughly – what it does • simlfy or extend by adding new elements • formulation few days, analysing months years

7. 21.2 Overview of Part IV • Chapter 22 • general strategies of analyzing ABMs • specific to ABMs • structural richness and realism • through controled simulation experiments • change assuptions submodels ..., statistical methods • Chapter 23 • sensitivity, uncertainty and robustness

8. Chapter 22: Analyzing and Understanding ABMs • 22.1 Introduction • 22.2 Example Analysis: The Segregation Model • 22.3 Additional Heuristics for Understanding ABMs • 22.4 Statistics for Understanding • 22.5 Summary and Conclusions

9. 22.1 Introduction • controlled experments • varying one factor at a time – effects on results • establishng causal relationships – understanding how the results are affected by each factor • scientific method – reproducable experiments • compleatly describing the model - lab or field • documenting • parameter values- input data- initial conditions • anaylxing results of experments

10. controlled simulation experiments • design, test and calibrate - models • understanding and analyzing what models do • How to analyze • model, the system and questions addressed, • experience and problem solving heuristics • Heuristics or rule of tumbs • often usefull but not always • not unscientific

11. Learning objectives • Understan purpose and goals of analyzing full AMBs • finished or preliminary • ten heuristics • statistical anaysis for ABMs

12. 22.2 Example Analysis: The Segregation Model • ODD • purpose • entities, state variables and scales • turtles – households • loaction, heppyness • houses - patches • space 51*51 • time • stop – all heppy

13. Processes • if all happy stop • for all housholds not happy • move • update heppyness • produce output • Design concepts

14. submodels • move • update

15. Analysis • #turtles 2000, %-similar wanted 30% • after about 15 ticks • average simularity of neihborhoods 70%

16. Heuristic: try extream values of parameters • model outcomes is often easy to predict or understand • Set tolernce low • Set tolarance high

17. Heuristics: find tipping points in model behavior • qualitatively diferent behavior at extream values of parameters • vary the parameter try to find “tipping point” • the parameter range – model behavior suddenly changes • regiems of control • process A after some point process B may dominent

18. Heuristics • try different visual representations of the model • color size patches • run the model step by srep • look at striking or strange patterns • at interesting points keep the parmeter and vary other parameters

19. 22.3 Additional Heuristics for Understanding ABMs • use several “currencies” for evaluating your simulation experiments • analyze simplified version of your model • analyze from the buttom up • explore unrealistic senarios

20. Heuristics: use several “currencies” for evaluating your simulation experiments • ABMs are rich in structure • “currincies” summary statistics or observations • emprical measures in the real system • Ex: population modeling • measure – population size, wealth, age • analyze time series of population size • even mena or range • good currincies – observation in ODD design concept • several currincies – how sensitive they are

21. statistical distributions • mean, standard deviation, range • distribution – normal, exponential • characteristics of time series • trend, autocorrolation time units to reach a state • measures of spatical distributions • spatial autocorrelation, fractile dimension • measures of difference among agents • how some charcetristics different, distributions • stability properties • network characteristics • clustering coefficient, degree,centrality, average path length

22. Heuristics: analyze simplified version of your model • simplfy • ABM so many foctors affect output • reduce complexity • undertand what mechnisms what cause what results • make the environment constant • make space homogenuous • all patches same over time • reduce stocasticity • fixed initial conditions – all agent alike • insteaad of randomness use mean values • reduce the system size • turn off some actions in model schedule • manually create simplified initail configrations

23. Heuristics: analyze from the buttom up • ABMs hard to understand • behavior of its parts – agents and their behavior • first test and undertsnd these • then full model • anaysis of submodels • developing theory for agnet bahavior

24. Heuristic: explore unrealistic senarios • simulate senarios – never occur in reality • to see direct effect of a process or mechanism on resutls – remove it • Ex 2: How investor behavior affects double –auction markets • interesting contrast: • models – unrealistically simple investor behavior • produce system level results not so unrealistic • conclusion • complex agent behavior – not reason for complex market dynamics • market rules themselfs might be important

25. 22.4 Statistics for Understanding • statistics – analysis and understanding • infer causal relatinships from a limited and fixed data • ABM – • generates as much data aa possible • additional mechnisms • if cannot explain • add new mechanizms • change assuptions • purpose and mind-set of • statistics and simulation modeling • are quite different

26. summary sttistics • aggregagting model outputs - mean, standard deviation • extream values might be importnat so outliers are usefull • Contrasting senarios • detect and quantify differences between senarios • assumptions may affect resutls – number of treatments • easier to change assuptions • t test ANOVA

27. Quantifying correlative relationships • regression, ANOVA • statistical relationsships between inputs – outputs • inputs: paramerters, initial conditions, time series • response surface methodology • not directly idenfy causal relations • but idenfity relavant factors • meta-models • Comparing model outputs to emprical patterns • calibration

28. 22.5 Summary and Conclusions • combine • reasoning, strong inference, systematic anaysis, intiution and creativity • once build an ABM or freeze it • understand what is does – controlled simulation experiments • heuristics • publications • heuristics in figure 22.3 • add your own heuristics

29. Chapter 23: Sensitivity, Uncertainty and Robustness Analysis • 23.1 Introduction and Objectives • 23.2 Sensitivity Analysis • 23.3 Uncertainty Analysis • 23.4 Robustness Analysis • 23.5 Summary and Conclusions

30. 23.1 Introduction and Objectives • Does an ABM reproduce observed patterns robustly • or sensitive to change in model • parameter • structure • how uncertain are model results • if model reproduce patterns for • parameters – limited range or values • key processes are modeled one exact way • unlikely to capture real mechanism underlying the patterns

31. testing and documenting the sensitivity of model ooutput to changes in parameter values is important: • 1 – how strongly the model represents the real world phenomena • 2 – helps to understand relative imprtance of model processes • high sensitivity to parameter – the process linked to that parameter controls model output and system behavior than other processes • high sensitivity to a parameter – need not be bad • diagnostic tool to understand models

32. Basic Definitions • Sensitivity analysis (SA) exercises how sensitive model’s outputs are to changes in parameter values • Uncertainty Analysis (UA) looks at how uncertainty in parameter values affect the relaibility of model results • Robustness analysis (RA) explores robustness of results and conclusions of a model to changes in its structure

33. Learning objectives • local SA with BehavioSpace • visualizations – SA with several parameters or global SA • stamdard UA methods with BehaviorSpace • steps of conducting RA

34. 23.2 Sensitivity Analysis • to perform SA • full version of the model • “reference” parameter set • one or two key outputs – “currencies” • controled simulation conditions • initial conditions • time series inputs • number of time steps

35. 23.2.1 Local Sensitivity Analysis • Objective – how sensitive the model • currency seleced • parameters one at a time • usually all parameters • Steps • range of parameter – +or-5% • run model for reference P and p-dP p+dp – replicate • mean C values • calculate sensitivity – approximatins to partial derivative

36. Three types of parameters • high values of S • processes important in the model • high value of S and high uncertainty in reference valus • little information to estimate their values • special attantion as calibration • target of emprical research to reduce uncertainty • low values of S • relatively unimportant processes - removable

37. Alternatives • only positive change • C’/C absolute change • distibuton of C – variance • diferent values of P • regression of C on P

38. Limitations • linear response so parameter change is small • parameter interractions missing • around reference parameter set

39. 23.2.2 Analysisof Parameter Interractions via Countour Plots • contour plots – interractions of two parameters • all other parameters are kept constant • Multi-panel contour figures – model sensitivity • many parameters at onces

40. 23.2.3 Global Sensitivity Analysis • vary all parameters over their full range • look at several currencies - understanding • “brute force” - analysis • for each parameter several values • replicaitons • hard to measure currencies • regression analyis – respose surface methods • design of simulation experiments • not all combination of parameters

41. 23.3 Uncertainty Analysis • similar to SA but • to understand how • the uncertainty in parameter values and • model’s sentitivity to parameters • interract to cause uncertainty in model results • parameters – measurment errors • steps of a UA • identify the parameters • for each parameter – define a distribution • belief or measurment errors • run the model many times – drawing from distributions • analyze distribution of model results

42. 23.4 Robustness Analysis • Weisberg (2006) • Whether the results depends on the • esentials of the model or • details of the simplfying assuptions • study number of distinct similar models of the same phenomena • despte different assumptions – similar results • robust theorm - relatively free of details of the model

43. modeling, PO • robust explanations of observed patterns • if a model’s ability to reproduce characteristic patterns of a real system is very sensitive to its details • it likely does not capture real mechanisms driving the real system

44. A full model – frozen • two heuristics: • analyze simplified versions • explore unrelistic senarios • but may look at more complex versions as well

45. General steps of RA • start with a well tested model version • decide which elements to modify • the way to initilize model entities – agents homogenous or not • processe in different ways – e.g. siplified or complex objectives for agents • test modified model – reproduce observed patterns

46. theory development – agent behavior • testing alternative submodels • RA • testing alternative versions • 23.4.1 Example: Robustness Analysis of the Breeding Synchrony Model • left as an exercise

47. 23.5 Summary and Conclusions