VPMM Evolved: An Update

1. VPMM Evolved: An Update S.Y. Harmon (541) 863-4639 harmon@zetetix.com Simone Youngblood (703) 824-3436 syoungblood@dmso.mil September 2007

2. Credibility, The Beginning of the VPMM • The motivation for the VPMM coalesced in 1999 when the ITOP Committee on V&V Procedures tried to develop levels of credibility & asked for help. • In response, Ms. Youngblood asked her team to explore simulation credibility & its relationship to V&V. • We started with the definitions of credibility: • Credibility is nearly synonymous with trustworthiness. • Credibility is a property of the information being presented. • However, credibility is a property that involves the belief of the observer. • Therefore, the perception of credibility is inherently subjective. • Credibility is only loosely coupled to the process for deriving the information. • Therefore, the integrity of the process can only contribute to credibility if the observer understands that process and appreciates its limitations (e.g. mathematical logic). • Further, the observer must also trust that the people who applied that process did so correctly. Observations on Credibility The quality of being trusted and believed in The quality of being convincing or believable Apple Dictionary, Version 1.01 The believability of a statement, action, or source, and the propensity of the observer to believe that statement Wikipedia The quality, capability, or power to elicit belief A capacity for belief The Free Dictionary The quality or power of inspiring belief Capacity for belief Merriam-Webster Online Dictionary

3. Proposed Levels of Validation Process Maturity(as of May 2006)

4. Proposed Levels of Validation Process Maturity(as of May 2006)

5. VPMM Application to Real Simulation Validation Efforts • Constructed an evaluation questionnaire that elicits information characterizing validation processes (36 questions) • Applied this questionnaire, through interviews, to 11 simulations • Analyzed the questionnaire responses to obtain the results in the table below The numbers in the leftmost cells are the maturity levels of each of the components of the validation process. The VPMM was also applied in formulating the content of the proposed IEEE Standard for the VV&A Overlay to the FEDEP.

6. VPMM Development & Application History

7. Lessons Learned from the VPMM’s Applications • Despite performing these evaluations in an interview setting, several important terms seemed to be misunderstood even in the VV&A community (e.g., validation criteria, referent, conceptual model & subject matter expert). • This version of the VPMM does not handle evaluation of efforts that do not use the information from a simulation conceptual model. • Most VV&A practitioners feel that verification improves the validation process but they do not verify against a conceptual model; they do not use verification products to guide results validation; and the verification products do not contribute to confidence estimation. What’s left? • Some simulation programs build sophisticated referents but appear not to use them to estimate error. Why? • Face validation can be improved with information from conceptual model validation and verification but this VPMM version does not resolve that improvement. • Use & V&V histories can contribute valuable information to the validation process but this VPMM version does not apply that information. • This version of the VPMM does not directly associate risk reduction with higher levels of maturity & account for improved ability to estimate validation effort costs.

8. Approach to Building the Next Generation VPMM • Construct a model of simulation use using decision theory, simulation-based decision making • Characterize the uncertainties, and therefore the risks, that a user of simulation-based decision making encounters • Develop a model of the VV&A processes needed to minimize, or at least identify, the risks that a user of simulation-based decision making encounters • Develop individual models of validation, verification and accreditation processes from the global VV&A model that take into account all of the information available to VV&A practitioners • Identify the functions associated with each of the components of risk-based VV&A • Organize a maturity matrix for risk-based VV&A using the same principles as the original VPMM that structures the risk-based VV&A functionality

9. Simulation Simulation situation model situation model Simplified Models of Simulation-Based Decision Making observed situation state user actions User Situation effects predictions actual situation state A decision maker uses a simulation to change decision making under uncertainty to decision making under either risk or certainty but depends upon the simulation to provide complete and correct information to make that change. VV&A Processes user needs accreditation recommendations observed situation state user actions User Situation effects predictions actual situation state VV&A processes can provide that decision maker with information about the conditions under which the simulation’s results are complete and correct enough for their intended use and thus reduce the risk that they take by incorporating the simulation into their decision making.

10. More About Use Risk & Decision Errors • Users can make one of two types of error when using simulation results in their decision making: • Type I error • The user chooses to ignore simulation results even though they are sufficiently correct and complete enough for their use. • The user incorrectly believes the simulation results to be incomplete or incorrect, may rely upon less reliable sources for the information to support their decisions, and make poor decisions as a result. • Example: Crater simulation used for Columbia damage assessment decision • Type II error • The user chooses to use simulation results that are insufficiently complete or correct enough for their use. • The user incorrectly believes the simulation results to be complete and correct enough to support their decisions and may make poor decisions as a result. • Example: Patriot control rule testing • Experiencing Type II errors from using a simulation may lead the user to make Type I errors. In risk-based VV&A, the accreditation recommendations define the conditions under which the user can depend upon a simulation’s results while incurring risks below tolerable levels.

11. Top Level Model of Risk-Based VV&A Use Risk Modeling (accreditation agent) user needs use characterization & prioritized validation criteria validated conceptual model simulation referent Simulation Verification (developer or V&V agent) Simulation Validation (V&V agent) use & V&V histories conceptual model verified conceptual model & integrated verification evidence simulation results V&V evidence Accreditation Analysis (accreditation agent) accreditation history accreditation recommendations In order to support risk-based VV&A, all of the components of the VV&A processes must focus upon contributing to defining the accreditation recommendations (i.e., the conditions of recommended use). These recommendations will enable the user to avoid making Type II errors and the credibility gained through that success will help them avoid making Type I errors as well.

12. Model of Use Risk Modeling need & risk elicitation discourse User Need & Risk Elicitation risk information needs use information needs needs information risk information Use Risk Modeling Requirements Modeling requirements model prioritized acceptability criteria use characterization Use risk modeling replaces the front end components of the accreditation assessment. The process now focuses upon eliciting user needs and what they know about their risks then elaborates upon that input using requirements and use risk modeling techniques. This risk modeling permits prioritizing acceptability criteria based upon risks.

13. Conceptual Model Verification verified conceptual model system conceptual model Design Verification system design products verified design products validated conceptual model Implementation Verification verified implementation products system implementation products implementation verification evidence Verification Analysis integrated verification evidence design verification evidence CM verification evidence Model of Risk-Based Verification Verification ensures the consistency of the conceptual model and contributes evidence to the validation process. This evidence reduces the coverage needed in results testing and increases the confidence that the simulation’s results are correct and complete enough for the intended use.

14. Model of Risk-Based Validation prioritized acceptability criteria use & V&V histories conceptual model use characterization History Evaluation Conceptual Model Validation verified conceptual model validated conceptual model relevant historical validity information Results Validation Planning integrated verification evidence results validation plan simulation referent Results Validation simulation results V&V evidence All available information, not just the conceptual model and verification products, contribute to focusing the results validation (through design of experiments) and, thus, building the picture of the simulation’s representational capabilities in the context of the intended use.

15. prioritized acceptability criteria use characterization V&V evidence V&V Evidence Integration accreditation history integrated suitability picture Use Risk Analysis Accreditation Recommendation Derivation accreditation recommendations use risk assessment Model of Risk-Based Accreditation Analysis The back end of the accreditation process builds an integrated picture of the simulation’s suitability for the intended use through evidence theory and estimates the confidence in that picture. This information contributes to both the use risk assessment and accreditation recommendations.

16. increasing use risk Maturity Characteristics of Validation-Related Products

17. Provable (5) Confident (4) Accurate (3) Complete (2) Subjective (1) Initial (0) Salient Properties of Validation Process Maturity Levels

18. Summary & Conclusions • The VPMM has gradually matured since 2002 through generous feedback from the technical community. The VPMM has been vetted by the • US DoD M&S community through the DoD VV&A TWG & DoD standards process, • VV&A community through Foundations ‘04), and • M&S community in SISO through SIWs & in SCS through JDAMS article. • Proof of concept studies are complete and the results from these studies will further mature the VPMM concepts. • Activities aimed toward creating a SISO standard have been started. These activities may integrate with a larger study group exploring the fundamentals of V&V. • Activities aimed toward creating an interim DoD standard have been started and draft documentation will be completed in July 2007. • The VPMM concepts are being expanded to encompass risk-based VV&A.