A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation

1. A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation Lieutenant Colonel Wayne Stilwell United States Army 7 September 2006 Dr. Donald E. Brown, Advisor Dr. William T. Scherer, Chair Dr. Stephanie Guerlain Dr. Paul Reynolds COL (Dr.) George F. Stone, III

2. Degree Requirements • Course work completed in May 2005 • Seminar in Honolulu, Hawaii 2-10 FEB 2006: Project Albert 11th International Workshop for Agent-Based Simulation • 13th PAIW in Netherlands 12-17 NOV 2006 • Will lead a team of researchers into command agent simulation calibration experimentation • Article submitted to: • Journal of Defense Modeling and Simulation “A Calibration and Validation Process (CAVP) for Complex Adaptive System Simulation • Planned Articles: • IEEE Journal (Proof adapted from Luenberger 1973) • MORS (Replication of a live experiment with agent-based simulation) • “Command Agent Calibration” - 13th PAIW

3. CAVP • CAVP is an iterative, information engineering-based process that calibrates CASS agent parameters to a range of acceptable outputs. • CAVP relies on: • Empirical data or Expert opinion on real system • Response surface methods (to include ERSM), data mining tools such as classification trees and linear regression, NOLH design of experiments, and expert opinion of reasonable input

4. Problem Statement • Simulation validation techniques do not currently offer an ability to: • Measure the influence of non-linear relationships that contribute to the outcome of a dynamic system • Reduce the complexity of higher order interaction • Calibrate multiple simulation inputs to desired outputs • Validate a CAS via the entire component comparison before white-box validation

5. Complex Adaptive System MOUT as a Complex AdaptiveSystem • Agent-based • Heterogeneous • Dynamic • Feedback • Organization • Emergence • Non-linear interaction • Non-reductionism • Emergent behavior • Hierarchical Structure • Decentralized Control • Self Organization • Non-equilibrium Order • Adaptation • Collectivist Dynamics

6. Literature Review Key Authors

7. Calibration Definition • The process of adjusting parameter values in the simulation model to better represent the underlying system • “Calibration” implies the existence of a standard to judge against.

8. Validation Definition (DMSO) • The quality of being inferred, deduced, or calculated correctly enough to suit a specific purpose. • The degree of validity is the level of trust a simulation user can place in the output of the model.

9. Classic Validation Concept • White Box first: Validate each module according to its components • Black Box next: Compare the total system output to actual system output

10. Literature Review Key Points • Aggregation based simulations are an improvement over differential equations-based simulations when modeling complex phenomena • CAS require more sophisticated validation methodologies than are currently available to improve the value of decisions • Behavioral input of each agent creates emergent behavior, requiring more extensive validation techniques • Statistical approaches like the ERSM can provide the basis for an improved validation method.

11. ERSM (Schamburg and Brown )

12. NOLH

13. The CAVP • Determine CAS for investigation • Examine extant system output • Determine Measures of Performance (MOP) • Develop Inputs • Construct the CASS • Determine a NOLH DOE • Compare MOP using Metrics of Evaluation (MOE) • Conduct Global Convergence Optimization on responses not in tolerance • Declare calibration state; If not calibrated, use CART to determine causality of inputs

14. CAVP Proof

15. Iterative Composite Mapping

16. The Experiment Recreate live soldier firefight CAS (blank rounds with sensors) via a CASS • 4 varying scenarios • 5 Measures of Performance (Blue casualties, red casualties, blue rounds fired, red rounds fired, time in seconds) • Metric of Evaluation (distance function) • Used MANA as simulation of choice • NOLH-based DOE, 33 design points, 200 iterations per design point. • Heterogenous soldiers • Expert opinion on input ranges for 10 control variables • Exogenous variables held steady throughout experiment

17. Analysis

18. Input Parameters

19. MOP Target Values

20. NOLH Design of Experiment

21. Sample Result d(Cij, Ej)≤Υj

22. Results Scenario 1 BLUE RDS TIME RED RDS RED CAS BLUE CAS d(Cij, Ej)≤Υj

23. CART Analysis of Red Rounds, Scenario 1 Classification Tree Regression Equation

24. Conclusions • MANA will calibrate three of the five MOPs • MANA is not valid for use unless rate of fire can be changed in the simulation • If the simulation can be changed, it may come into calibration for the final two MOPs, and then could become a valid CAS

25. Contributions • A process to calibrate agent input against an error tolerance for complex adaptive system simulations • A simulation validation methodology that uses a reverse order from classical validation methodologies • A composite-mapping process that efficiently searches a problem space and guides a simulation developer towards more effective simulations

26. CAV Process Conclusions • Relates CASS output back to agent inputs and can effectively calibrate a simulation • Determines the calibration state of agents, and determines the validation state of the CAS • Can guide the modeler and inform the simulation development process