Verification, Validation and Accreditation of Agent-Based Simulations

1. Verification, Validation and Accreditation of Agent-Based Simulations Deborah Duong

2. Purpose • To introduce Agent-Based Simulation • To propose measures of effectiveness for Agent-Based Simulation

3. What is an Agent-Based Simulation? • “Agent-Based Simulation” (ABS) is broadly defined • An ABS is a simulation in which entities have “agency” • Agents can perceive and behave in their environment based on goals • Agent-Based Simulation is used for modeling living systems • Biological and social systems • Non-living systems are mindless, and therefore don’t have “agency” • The concept of “emergence” is important • Agents behave according to one set of rules • New patterns “emerge” from individual behaviors • Emergence is micro-macro integration

4. How does Agent-Based Simulation Compare ? • Other methods that don’t involve agency or minds are also used to describe living systems • Discrete Event Simulation • Events of a process are scheduled to occur at discrete points • System Dynamics Simulation • Looks at the flow of “fluid” levels over time • Time delays are important • Social Networks • Patterns in the arrangement of entities to each other are important • These methods are at their best when modeling “non-mental” phenomena • Ecology • Predator-Prey cycles • The Economy • Cycles not based on “beliefs” (like the stock market is) • Any time entities act similarly • Everybody eats! • Non-agent simulation methods model flows and arrangements of “averaged” entities • Their “State” does not change, because entities are not modeled explicitly • They are not “networked” • They are viewed from an external, “etic” standpoint

5. Why some Computational Social Scientists prefer ABS • Their preference depends on their feelings on the importance of “agency” and minds • They may believe that other tools are not as rich • Other tools tend to make “heroic assumptions” • They often can not model the crux of the problem • They are more descriptive than causal • North and Macal: • “We believe that in the future virtually all computer simulations will be agent-based because of the naturalness of the agent representation and the close similarity of agent models to the predominant computational paradigm of object-oriented programming.”

6. Cognitive vs. Reactive Agents

7. Data-Based vs. Theory-Based ABS

8. Agent Based Simulation and VV&A • Verification • Determination of whether a simulation expresses a theory well • Validation • Determination of whether a simulation has fidelity with the real world • Accreditation • Determination that a simulation is useful for analysis of a particular domain • Verification, Validation and Accreditation of agent based models is problematic • VV&A originated in physics models • The nature of social science has implications for agent based VV&A

9. Agent-Based Simulation and Verification • The more a simulation has the power to express a theory, the more the simulation is verified • A System Dynamics model of a verbal theory wouldn’t have a high degree of “verification” unless that theory was about time-delays • The referent of any mathematical or simulation model is a theory • In physics based models, verification is “doable” • In physics-based models, verification is mainly about bugs • Replication, or using a different method to simulate the same theory, can help debug agent based social models • In social-science based agent models, verification is the central issue • Verification is about technology to represent an idea • Newton had the technology of the calculus • The technology to simulate social theories is not trivial • For example, a social theory about human learning may need a computer that can match a human in learning • With knowledge of available tools and creativity, Verification is just a matter of good (scientific) taste, for now

10. The Social Literature as the Referent • Fitting raw data is not enough for verification • Data can be over-fitted • One could “simulate” by never addressing cause, by only making correlated things appear magically • Since “why” is not modeled, the simulation is not generally applicable • If it wont model a new situation, it wont model itself well either • If there are no causes a level under the phenomena you model, you are only describing, not analyzing • You can not explore the new levers to change outcomes, other than the ones you put in the simulation to begin with • Data should be fitted through a theory of social science • Thoughtful models in the social literature are preferred to models from other fields • Just because we have the tools to describe time delays, physical phenomena, and epidemiology doesn’t relate them to social theory • Knowledge of all tools is needed to model the richness of the social world • Tested by: surveying the relative frequency of issues in the social literature and comparing to the relative frequency of issues in an ABS

11. Agent-Based Simulation and Validation • The more explanatory power an agent-based simulation has, the more the simulation is validated • A simulation model should match the data in the world in the way that its theory matches it • Validation of agent based simulation is dependant on verification: If an agent based simulation is not first verified, it will not be valid • Validation of agent based simulation is dependant on the explanatory power of its referent theory as well • Technology that enables verification enables exploratory creation of theories with explanatory power

12. What can we expect from an ABS? • To address validation, let us ask, what can we expect from a theoretically perfect ABS? • Even if the agent based model was completely correct, it still could not do long term prediction • The social world is full of “Schelling Points”: arbitrary phenomena • We can expect it to display similar patterns to the real world, but not the exact data of the real world • It should have the same correlative patterns • Links between events in a simulation should have a similar strength to links between corresponding events in the real world • It should develop a distribution of plausible results similar to the real world • Tested by “separating the test set from the training set” • It should be able to make a short term prediction of “types” of phenomena • A live connection to data is essential • An agent based simulation *is* a theory • It is a theory represented in a form amenable to computation • The theory that best matches the (patterns in) data is the best theory

13. Validating Agent-Based Simulations • Data-Based vs. Theory-Based Agent models: How do we simulate both theory and data well? • The trajectory of a theory-based simulation can be made to pass through particular data • Random number massaging • Co-evolutionary “seeding” • Because the data emerges from the simulation itself, it models the next state better • It is validated if it models not only patterns in data, and the social literature well, but it also models causation well • Ockham’s razor: If many known phenomena emerge from a few known phenomena, you have modeled a cause well

14. Agent-Based Simulation and Accreditation • Rating for a usage in a domain is based on correctness of past usage in that domain • Pattern-based correctness • Social Science simulations are so complex, that scientific insight is needed in each new application • There is no way to generalize what tool will always be good in advance for what domain • Accreditation efforts should be devoted to confirming that a simulation does have expressive and explanatory power after the tool is chosen for the application • When is a model ready for use in analysis? • When it predicts patterns in data and the occurrence of “types” of events consistently when given new data

15. Myths of Agent-Based VV&A • “Chaos theory says there is no order, and any small change makes a big change in the outcome” • The social world is full of order and homeostasis • “The cause of emergent phenomena is so complex that it is unknowable” • Cause is knowable because it is contained “in the box” • Scientific experiments can tease out cause • Computer experiments can “hold all else the same” better than real world experiments can • Statistics can find cause in Monte Carlo ABS

16. Implications for Existing VV&A Techniques • Exploratory Space and Risk Analysis • Testing simulations at the boundaries where it matters • Nonlinearities in agent-based simulation means we don’t know where it matters • “Agency” can be taken advantage of in strategic data farming • Bottom-up VV&A • Making sure that the lower level is VV&A’d and that will take care of the upper level • But you don’t know what to emphasize in the lower level until after the emergence happens

17. Summary • Agent Based Simulations model “Agency” • ABS are best used when mental processes and dynamic networks are important • ABS may be typed according to two dimensions • Cognitive/Reactive • Data-Based/Theory-Based • There is hope for Agent Based Simulation Verification, Validation and Accreditation • We have ways to measure • Similar patterns to the real world correlative data • Match to the social theory in literature • Explanatory power (Ockham's Razor)