Agent-Based Modeling: Verification, Validation, and Replication

1. MIS 585 Special Topics in MIS: Agent-Based Modeling 2015/2016 Fall Chapter 7 Verification, Validation, and Replication

2. Outline Correctness of a Model Verification Validation Replication Summary

3. Verification, Validation, and Replication • Previous chapters • importance and utility of ABM • extend an existing and build a new ABM • components of an ABM • collect and analyze results from an ABM • In this chapter • correctness and usefulness of an ABM • how an implemented ABM corresponds to a conceptual model • how to assess the match between an ABM and the real world

4. Outline Correctness of a Model Verification Validation Replication Summary

5. Correctness of a Model • Model accuricy can be evaluated through three modeling processes • verification, validation, and replication • Validation: • whether the implemented model corresponds and explains some real world phenomena • Verification: • implemented model corresponds to the target – conceptual model • Replication: • implementation of a conceptual model of another researcher

6. Outline Correctness of a Model Verification Validation Replication Summary

7. Verification Communication Describing Conceptual Models Verification Testing Beyond Verification Sensitivity Analysis and Robustness Verification Benefits and Issues

8. Verification • goal of verification • elimination of bugs in the code • General guidline - Basic modeling principle • start simple • verificaiton is easy as well • adding components incrementally towards the RQ • other compleications • interaction between components

9. Voting Behavior Model (VBM) • political scientists • people’s social interactions determine voting in elections • based on polls and election results • people inital way of wote • they talk to neigbors and friends discuss the way of voting may change their vore • may happen several times during an election period

10. Communication • model author and implementer • may be different • varification is more difficult • get familize • implemneters – domain • domain experts – modeling tools • conceptual model – implemented

11. Describing Conceptual Models • flowcharts • rounded squares – start and stop • squares – initiate processes • dimonds – decision • pseudo-code • Unified Modeling Language (UML) • ABM specific concepts to UML • ODD

12. Pseudo-code of the VBM for each voter: set vote either 0 or 1 with equal probabiity loop until election if majority of neighbors’s votes 1 and votrers vote is 0 set vote 1 elseif majority of neighbors’s votes 0 and votrers vote is 1 set vote 0 if vote = 1, eet color blue else set color red Display count of voters with 1 Display count of voters with 0 end loop

13. Verification Testing • Implementing the pseudo-code in NteLogo • then, write a test procedure for testing whether the number of green and blue patches are roughly equal or not

14. setup patches-own [ vote ;; my vote (0 or 1) total ;; sum of votes around me ] to setup clear-all ask patches[ if random 2 = 0 ;; half a chance of this [ set vote 1 ]

15. setup (cont.) ask patches[ if random 2 = 0 ;; half a chance of this [ set vote 0 ] ask patches [ recolor-patch ] check-setup reset-ticks end

16. recolor-patch to recolor-patch ;; patch procedure ifelse vote = 0 [ set pcolor green ] [ set pcolor blue ] end

17. check-setup ;; This procedure checks to see if the SETUP procedure sets up the model with roughly ;; equal numbers of blue and green patches to check-setup ;; count the difference between the number of green and the number of blue patches let diff abs (count patches with [ vote = 0 ] - count patches with [ vote = 1 ]) if diff > .1 * count patches [ print "Warning: Difference in initial voters is greater than 10%." ] end

18. corrected code patches-own [ vote ;; my vote (0 or 1) total ;; sum of votes around me ] to setup clear-all ask patches [ set vote random 2 ;; set vote to either 0 or 1 recolor-patch ] reset-ticks check-setup end

19. Unit Testing • (component testing) • writing small tests to check whether individual units are working correctly in the code • “in model” unit testing – “online” unit testing • “offline” unit testing: seperate piece of code with input values

20. go to go ask patches [ set total (sum [ vote ] of neighbors) ] ;; use two ask patches blocks so all patches compute "total" ;; before any patches change their votes ask patches[ ifelsevote = 0 and total >=4 [ set vote 1 ] [ if vote=1 and total <= 4 [ set vote 0 ] ] check-setup recolor-patch ] tick end

21. Beyond Verification • no equilibrium • jaggled edges and cyclses • tied votes • voters on the edge cycles back and forth • add a switch CHANGE-VOTE-IF-TİED? • when voters neighbors are closely divided, vote underdog • not for majority

22. go to go ask patches [ set total (sum [ vote ] of neighbors) ] ;; use two ask patches blocks so all patches compute "total" ;; before any patches change their votes ask patches [ if total > 5 [ set vote 1 ] if total < 3 [ set vote 0 ] if total = 4 [ if change-vote-if-tied? [ set vote (1 - vote) ] ] ;; invert the vote

23. go (cont.) if total = 5 [ ifelse award-close-calls-to-loser? [ set vote 0 ] [ set vote 1 ] ] if total = 3 [ ifelse award-close-calls-to-loser? [ set vote 1 ] [ set vote 0 ] ] recolor-patch ] tick end

24. Sensitivity and Robustness • how sensitive the outputs results of a model to changes in • parameters, initail conditions, environments, behaviorsl asumptions • examination of the impact of varying model parameters on results

25. Sensitivity and Robustness • how sensitive the outputs results of a model to changes in • parameters, initail conditions or other asumptions • E.g.: in the VM • how the final voting behavior sensitive to initial distrfibution of votes • change initial distribution of green and red agents by a parameter

26. stop the model • no voter change votes or • 100 ticks • verification of new initialization • 10 repitations • graph initial distrbution vs. average count of votes • initial green patches vs. final green patches • Figure 7.6 • nonlinear effect in either side

27. depends on the output • a qualitative measure – • solid islands of both colors form • insensitive to change in initial distribution • Chagne environments • square to hex latice, network

28. Verification: Benefits and Issues • understanding unexpected outcomes • impact of changing parameters or rules • omceptual model – implemented model

29. Verification is difficult • chalanging to determine – surprising reults • a bug in code • outcome of a low level rule • conceptual model is wrong

30. not just verified or not • but continuous less or more verified • more tests • more sensitivity analysis • replication of the model

31. Outline Correctness of a Model Verification Validation Replication Summary

32. Validation Macrovalidation vs. Microvalidation Face Validation vs. Emprical Validation Validation Benefits and Questions

33. Validation • process of ensuring that there is a correspondence between the the implemented model and the reality • by its nature • complex, multilevel and relative • RQ in mind • validate aspects of the model related to RQs

34. Two Axes of Validation • two axes Rand and Rust 2011 • 1 - level at which validation process is occuring • microvalidation: behviors and mechanisms of agents are match up with real world analogs • macrovalidation: ensuring that the aggreage, emergent preperties of the model is corresopeding with that ofreal world’s

35. Two Axes of Validation • 2 – level of detail of the validation process • face validation: shows that mechansms and preperties of the model looks like the same as that of the real world • Emprical validation: make sure that the model generates data that can be demonstraed to similar to the corresponding pateterns in the real world

36. Illustration • Flocking model (FM) in biology section of NetLogo library • regenerate patterns of birds focking • an ABM based on Boids models of Reynolds 1987 • birds flocks arise without being led by a special leader • each bird following exactly same set of rules from which flocks emerge • alignment, seperation, cohesion

37. the three rules • alignment: • a bird turns so that it is moving in the same direction as nearby birds • seperation: • a bird turns to avoid hitting another bird • overrides the other two: • if two birds are approaching each other, they seperate • cohesion: • a bird turns other nearby birds • attect heading, speed of birds constant • robust: swarming of insects, schooling of fish, V shaped flocking patterns of geeses

38. setup turtles-own [ flockmates ;; agentset of nearby turtles nearest-neighbor ;; closest one of our flockmates ] to setup clear-all crt population [ set color yellow - 2 + random 7 ;; random shades look nice set size 1.5 ;; easier to see setxy random-xcor random-ycor ] reset-ticks end

39. go to go ask turtles [ flock ] ;; the following line is used to make the turtles ;; animate more smoothly. repeat 5 [ ask turtles [ fd 0.2 ] display ] ;; for greater efficiency, at the expense of smooth ;; animation, substitute the following line instead: ;; ask turtles [ fd 1 ] tick end

40. flock to flock ;; turtle procedure find-flockmates if any? flockmates [ find-nearest-neighbor ifelse distance nearest-neighbor < minimum-separation [ separate ] [ align cohere ] ] end to find-flockmates ;; turtle procedure set flockmates other turtles in-radius vision end to find-nearest-neighbor ;; turtle procedure set nearest-neighbor min-one-of flockmates [distance myself] end

41. alignment to align ;; turtle procedure turn-towards average-flockmate-heading max-align-turn end to find-flockmates ;; turtle procedure set flockmates other turtles in-radius vision end to-report average-flockmate-heading ;; turtle procedure ;; We can't just average the heading variables here. ;; For example, the average of 1 and 359 should be 0, ;; not 180. So we have to use trigonometry. let x-component sum [dx] of flockmates let y-component sum [dy] of flockmates ifelse x-component = 0 and y-component = 0 [ report heading ] [ report atan x-component y-component ] end

42. seperation to separate ;; turtle procedure turn-away ([heading] of nearest-neighbor) max-separate-turn end

43. coherence to cohere ;; turtle procedure turn-towards average-heading-towards-flockmates max-cohere-turn end to-report average-heading-towards-flockmates ;; turtle procedure ;; "towards myself" gives us the heading from the other turtle ;; to me, but we want the heading from me to the other turtle, ;; so we add 180 let x-component mean [sin (towards myself + 180)] of flockmates let y-component mean [cos (towards myself + 180)] of flockmates ifelse x-component = 0 and y-component = 0 [ report heading ] [ report atan x-component y-component ] end

44. help procedures to turn-towards [new-heading max-turn] ;; turtle procedure turn-at-most (subtract-headings new-heading heading) max-turn end to turn-away [new-heading max-turn] ;; turtle procedure turn-at-most (subtract-headings heading new-heading) max-turn end ;; turn right by "turn" degrees (or left if "turn" is negative), ;; but never turn more than "max-turn" degrees to turn-at-most [turn max-turn] ;; turtle procedure ifelse abs turn > max-turn [ ifelse turn > 0 [ rt max-turn ] [ lt max-turn ] ] [ rt turn ] end

45. Macrovalidation vs. Microvalidation • comparing actual and model generated data at different levels • agent level – microvalidation • macro/system level – macrovalidation • E.g.: in FM • ask agents has same behavior or properties as real birds • loocation, heading, flockmade, vision • real birds many other properties • age, hungryk or not,.. • not relevant to the RQ

46. lRQ: whether agent fly like real birds so • lccation, heading, vision • 2D or 3D • Global properties of model and real birds • in ABM agent interactions leads to global or emergent properties • does the flocking patterns of model corresponds to reality

47. micro, macro or many levels in beteen • individual birs • flockmades reacts to each other • flocks as a whole • in equation-based modeling only macrolevel

48. Face Validation vs. Empirical Validation • ABM – reality • not unique – many good models • Face validation • look the model “on face” • conatins components behavior mechanisms corresponding real onces

49. Empirical validation • data generated by model corresponds to the data obtained from raal phenomena • measures and numerical data • Statistical data analysis • problem: defining inputs and outputs in real world may be dificult

50. Example FM • for Face validation • birds • micro level: • move in streight line, change directions, decisions based on local information • no complex calculations • macro level: • flocks at the macro level are compared • Emprical validation • angular momentum, size, velocity of flocks • comparable with real measures • depending on parameters and initial conditions model can generate data similar to real ones